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Overview 



Since 2002, New York City has closed more than 20 underperforming public high 
schools, opened more than 200 new secondary schools, and introduced a centralized high school 
admissions process in which approximately 80,000 students a year indicate their school pref- 
erences from a wide-ranging choice of programs. At the heart of these reforms he 123 new 
“small schools of choice” (SSCs) — small, academically nonselective, four-year public high 
schools for students in grades 9 through 12. Open to students at all levels of academic achieve- 
ment and located in historically disadvantaged communities, SSCs were intended to be viable 
alternatives to the neighborhood high schools that were closing. 

SSCs are more than just small. They were authorized through a demanding competitive 
proposal process designed to stimulate innovative ideas for new schools by a range of stakehold- 
ers and institutions, from educators to school reform intermediary organizations. The resulting 
schools emphasize strong, sustained relationships between students and faculty. Each SSC also 
received start-up funding as well as assistance and policy protections from the district and other 
key players to facilitate leadership development, hiring, and implementation. 

The first step in New York City’s high school admissions process is to require eighth- 
graders to select in rank order of priority up to 12 high schools that they want to attend; when an 
SSC has more applicants than spaces, the district uses a lottery-like process to randomly assign 
students to the SSC or to another school in the district. These lotteries provide the basis for an 
unusually large and rigorous study, supported by the Bill & Melinda Gates Foundation, of the 
effects of SSCs on students’ academic achievement. 

This report presents encouraging findings from that study, providing clear and reliable 
evidence that, in roughly six years, a large system of small public high schools can be created and 
can markedly improve graduation prospects for many disadvantaged students. Specifically: 

• By the end of their first year of high school, 58.5 percent of SSC enrollees are 
on track to graduate in four years compared with 48.5 percent of their non- 
SSC counterparts, for a difference of 10.0 percentage points. These positive 
effects are sustained over the next two years. 

• By the fourth year of high school, SSCs increase overall graduation rates by 
6.8 percentage points, which is roughly one-third the size of the gap in gradu- 
ation rates between white students and students of color in New York City. 



SSCs’ positive effects are seen for a broad range of students, including male 
high school students of color, whose educational prospects have been histori- 
cally difficult to improve. 
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Preface 



The traditional large high schools that typify so many school districts in this country 
— particularly our poor urban centers — are a relic of a former time, with too many of them 
characterized by shockingly high dropout rates and large numbers of young people who grad- 
uate unprepared for college-level studies. Despite much experimentation, little concrete evi- 
dence has emerged about how to turn around our lowest-performing public schools and equip 
America’s high school students with the skills they’ll need in today’s rapidly changing world. 

In New York City, however, a remarkable transformation now appears to be taking 
place. Since 2002, the city has closed more than 20 underperforming public high schools, 
opened more than 200 new secondary schools, and introduced a centralized high school admis- 
sions process in which approximately 80,000 students a year indicate their school preferences 
from a wide-ranging choice of programs. At the heart of these reforms lie 123 small, academi- 
cally nonselective public high schools for students in grades 9 through 12. These “small schools 
of choice” (SSCs) — a name coined by the authors of this report to highlight the fact that stu- 
dents at any academic level could choose to attend them — are located in historically disadvan- 
taged communities and were intended to be viable alternatives to the neighborhood high schools 
that were closing. This report presents the findings of the first large-scale, rigorous evaluation of 
that reform effort. 

What was the exact nature of the reform? It was rooted in the small schools movement, 
but it went further. SSCs are more than just small. They were authorized through a demanding 
and competitive proposal process that was designed to encourage and enable a range of on-the- 
ground stakeholders with innovative ideas — from educators to school reform intermediary or- 
ganizations — to start new schools. The result was an emphasis on features that offered support 
to disadvantaged and traditionally underserved students, such as reduced teacher load and 
common planning time as a way to ensure that all students were known well and to promote 
strong, sustained relationships between students and faculty. Each SSC also received start-up 
funding as well as assistance and policy support from the district and other key players to facili- 
tate leadership development, hiring, and implementation. In short, these schools were the prod- 
uct of a bottom-up, not a top-down, process. 

MDRC’s unusually large and rigorous study takes advantage of a lottery-like system 
that New York City uses to assign students when the high schools they choose are oversub- 
scribed. The findings show that it is possible, in a relatively short span of time, to replace a large 
number of underperforming public high schools in a poor urban community and, in the process, 
achieve significant gains in students’ academic achievement and attainment. And those gains 
are seen among a large and diverse group of students — including students who entered the 
ninth grade far below grade level and male students of color, for whom such gains have been 
stubbornly elusive. 
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While debates continue over test score differences and whether they can accurately pre- 
dict progression through high school and success later in life — despite little compelling evi- 
dence that scores alone can be relied upon to make such predictions — the reform effort that is 
the subject of this report has led to actual improvements in measures that point directly to in- 
creased attainment, graduation rates, and college-readiness: increases in attendance rates, in the 
number of credits earned from grade to grade over four years of high school, in high school 
graduation rates, in earning the New York State Regents diploma, and in achieving Regents 
scores in English that enable entry into the City University of New York. If the quality of the 
evidence presented here is rare, the results are rarer still. No comparable evidence has been pro- 
duced to date for any other major educational reform effort. 

Notably, New York City’s reform effort represented a partnership among a diverse 
group of people and agencies: Mayor Michael Bloomberg, Schools Chancellor Joel Klein, the 
NYC Department of Education, a consortium of philanthropies, the teachers and principals un- 
ions, nonprofit intermediaries, and community groups. It took enormous courage and convic- 
tion, and years of unrelenting toil, for this group of people with diverging perspectives to tackle 
the problem of failing high schools. The logistics alone of simultaneously closing and opening 
schools at this scale are daunting to contemplate, making the results all the more impressive. 

With the nation’s attention focused squarely on turning around failing urban high 
schools, this study demonstrates that it is possible to achieve meaningful changes at scale within 
a large, urban public school system. We look forward to following the story of the students in 
New York City’s small schools of choice to leam whether these gains grow as additional co- 
horts of students progress through their final year of high school, and whether the gains translate 
into success in postsecondary education and the labor market. 

Gordon Berlin 
President 
MDRC 
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Executive Summary 



Over the last decade, New York City has been the site of a systemwide high school 
reform effort that is unprecedented in its scope and pace. Since 2002, the school district has 
closed more than 20 failing high schools, opened more than 200 new secondary schools, and 
implemented a centralized high school admission process in which approximately 80,000 
students a year indicate their school preferences from a wide-ranging choice of programs. 

At the heart of these reforms lie the new schools that in this report are called “small 
schools of choice” (SSCs) — small, academically nonselective, public high schools that were 
opened between 2002 and 2008. Serving approximately 100 students per grade in grades 9 
through 12 and open to students at all levels of academic achievement, the SSCs in this study 
were created to serve the district’s most disadvantaged and historically underserved students. 
Prior to the 2002-2003 school year, these students would have had little option but to enroll in 
one of the city’s large, zoned high schools when they made the transition from eighth to ninth 
grade. Many of the large schools were low-performing, with graduation rates below 50 percent. 

This report presents encouraging findings from an unusually large and rigorous study, 
supported by the Bill & Melinda Gates Foundation, of the effects of SSCs on students’ academ- 
ic achievement in high school. It emerges at a moment when policymakers, practitioners, and 
researchers have identified the high school years as the point of greatest weakness within the 
education pipeline. The rationale for this collective focus is clear: far too many students drop 
out of high school, and the consequences of entering adult life without a high school diploma 
are increasingly grave. Amid a national call for change and a dearth of effective responses, the 
findings presented in this report provide clear and reliable evidence that: 

• In roughly six years, it is possible to create a large system of small public 
high schools that markedly improve graduation prospects for many of the 
disadvantaged students who choose to attend these schools. 

• In the schools being evaluated, positive effects on students’ progress toward 
high school graduation become apparent as early as the ninth grade and are 
sustained during the next two years; by the end of four years of high school, 
these effects culminate in higher rates of graduation. 

• These positive effects are experienced by a broad range of students who dif- 
fer in terms of their demographic characteristics, economic circumstances, 
and academic preparation. It is particularly noteworthy that the benefits of 
small schools extend to male high school students of color, whose education- 
al prospects have been historically difficult to improve. 
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This executive summary describes these findings and identifies their key implications 
for policy, practice, and knowledge-building. 



What Are Small Schools of Choice? 

The New York City public school system is the largest in the United States, with over 
1.1 million students enrolled in more than 1,600 schools. Over the past decade, it has been the 
site of an ambitious effort to reform the high school system, of which the creation of SSCs was 
a central part. Beginning in 2002, the New York City Department of Education (DOE) accel- 
erated and expanded efforts that had been under way since the mid-1990s to close large, low- 
performing schools and open new small schools in their stead. These reform efforts were 
supported by a consortium of funders led by the Bill & Melinda Gates Foundation — which 
ultimately invested over $150 million in New York City 1 — and were implemented in partner- 
ship with the teachers and principals unions. 2 The resulting changes in the high school land- 
scape transpired with unprecedented scale and rapidity. By 2008, 23 high schools with gradua- 
tion rates below 45 percent had been targeted for closure, and 216 new small schools, of which 
123 were SSCs, had been opened. 

While the district established a variety of small school models (shown in Box ES.l), 
ranging from transfer schools designed to serve students who had struggled in conventional 
high schools to specialized schools intended to serve the district’s highest-performing students, 
the predominant model was the small school of choice, 3 which, notably among the other school 
types, was academically nonselective and small not only in size but also in function. That is, 
structures such as reduced teacher load and common planning time (in which teachers meet 
together to discuss their students’ progress and problems) were recommended to ensure that all 
students were known well and to promote strong, sustained relationships between students and 
teachers. SSCs also had four other essential features: 

• SSCs were predominantly located in disadvantaged communities whose neighbor- 
hood high schools were closing. 



The Gates Foundation supported the DOE’s new school creation efforts in partnership with the Carnegie 
Corporation of New York and the Open Society Institute, and other systemwide initiatives benefited from at 
least $230 million worth of funding from philanthropies including the Wallace Foundation, the Michael & 
Susan Dell Foundation, and the Eli and Edythe Broad Foundation. Quint, Smith, Unterman, and Moedano 
(2010) provides a history of small schools in New York City, including the efforts undertaken by New Visions 
for Public Schools — which launched the New Century High Schools Initiative — that immediately preceded 
and served as the model for the school creation efforts under the Bloomberg/Klein administration. 

2 New Visions for Public Schools (2005). 

’“Choice” in “small schools of choice,” a term coined by the researchers, is meant to emphasize the fact 
that these nonselective schools are accessible to students of all academic levels. 
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New York City Small Schools of Choice 
Box ES.1 

Types of New Small Schools Opened Between 2002 and 2008 

General high schools offer a standard core curriculum in addition to elective courses and 
serve students at various levels of academic ability in grades 9-12. 

• Small schools of choice (SSCs) are both small and academically nonselective (123 
opened by the 2008-2009 school year). 

• Other general high schools are small and academically selective (38 opened by the 
2008-2009 school year). 

Transfer schools are small, personalized, full-time schools designed to help overage and 
undercredited students overcome obstacles to graduation (21 opened by the 2008-2009 
school year). 

Middle/high schools, typically serving grades 6-12 or 7-12, are intended to support stu- 
dents’ transition from middle to high school by enabling them to maintain relationships 
with familiar staff members and stay within familiar surroundings (33 opened by the 2008- 
2009 school year). 

Specialized high schools serve students who are high-performing academically and/or artis- 
tically. Admission usually depends on the student’s score on the Specialized High Schools 
Admissions Test, taken during eighth grade (1 opened by the 2008-2009 school year). 



SOURCE: MDRC calculations use High School Application Processing System data from 2004- 
2005 to 2007-2008 and school-level administrative records provided by the DOE for the 2002-2003 
to 2008-2009 school years. 



• SSCs were established via a demanding and competitive proposal process 
that emphasized the common design principles of academic rigor, personali- 
zation, and community partnerships. This process required a prospective 
school leadership team to articulate an educational philosophy and demon- 
strate how it would motivate teachers, community members, and partner or- 
ganizations around it. Additionally, the new school leadership had to develop 
a viable improvement strategy from the ground up. 

• SSCs benefited from an infusion of outside resources: new principals and 
teachers, partnerships with intermediary organizations that had expertise in 
starting new schools, and start-up funding from the district and its philan- 
thropic partners. 
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• SSCs received policy protections during their start-up period, including 
opening with only one founding grade of students (ninth grade) and having 
access to supports to facilitate procurement and hiring — such as special 
training for school principals and teachers; an amendment to the collective 
bargaining agreement, which gave principals more hiring discretion; and the 
conversion from a management system of regional offices to one in which 
schools had greater control over their budgets and educational programs. 



How Was the Study Conducted? 

In the spring of 2004, the city introduced the High School Application Processing Sys- 
tem (HSAPS), a centralized choice process that was to govern the placement of all entering 
ninth-grade students. HSAPS uses an objective, computer-based process to assign about 72,500 
entering ninth-graders annually to about 400 public high schools. 4 When they are in the eighth 
grade, students who participate in HSAPS indicate, in order of preference, up to 12 high schools 
they would like to attend. Each year, some schools have more applicants than seats available. 
When this occurs at an SSC, a lottery is created within HSAPS that randomly determines which 
students are assigned to that school. 

The analysis presented in this report uses data from the high school admissions process 
to identify a sample of students who chose SSCs, but who — because their chosen SSC had 
more applicants than seats available — were assigned via lottery either to that school or to a 
subsequent choice on their list. The analysis includes four annual cohorts of students who 
entered high school in the fall of 2005, 2006, 2007, and 2008, respectively — a total of 2 1,085 
students who applied to the 105 SSCs that were oversubscribed, and for which lotteries were 
held, during the study period. 

The existence of these lotteries provides an unprecedented opportunity to launch a 
rigorous study of the effects of this group of schools on student academic achievement, because 
the lotteries create two randomized groups among students who chose a given SSC — those 
who won its lottery and were assigned to the SSC and those who lost its lottery and were 
assigned elsewhere. Future outcomes for these two groups can be compared to obtain valid 
estimates of the effects of SSCs on student achievement. The lotteries created by HSAPS 
together with the unusually large size of the randomized sample they produced allow for a high 
degree of validity and precision in the present analyses. Thus, one can have considerable 



4 Although approximately 80,000 students participate in HSAPS each year, a small percentage of those 
students do not receive a match and advance to high school through a borough enrollment office instead of 
through HSAPS. Thus, an average of 72,500 students are assigned through HSAPS. 
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confidence in them. Using these lotteries as the basis for its analysis, this report presents the 
estimated effects of enrolling in a small school of choice versus enrolling in one of the other high 
schools that are available to the average incoming ninth-grader. 5 

Most of the schools attended by students who did not enroll in an SSC were older and 
larger than the SSCs: all SSCs were created since 2002 while two-thirds of the schools attended by 
the non-SSC enrollees were established before then, and the ninth-grade classes averaged 129 
students in SSCs and 635 students in the non-SSC schools. 6 However, it is important to remember 
that the SSCs are not being compared with the large, failing schools they replaced but rather with a 
wide range of schools that were also operating in a reform-rich atmosphere. 



What Are the Effects of Small Schools of Choice? 

Making a successful transition into high school is a critical step toward graduation. For 
example, the Consortium on Chicago School Research found that high school students who are 
on track to graduate by the end of their first year — meaning that they have earned at least 1 0 
credits and are failing no more than one core subject — are three and a half times more likely to 
graduate in four years than are other students. 

The First Three Years of High School 

SSCs have a substantial positive impact on the transition into high school during ninth 
grade, according to data using all four cohorts (see Table ES.l): 

• SSC enrollees were 10.8 percentage points more likely than the students who 
enrolled in other schools to earn 10 or more credits during their first year — 

73.1 percent compared with 62.3 percent. 

• SSC enrollees were 7.8 percentage points less likely to fail more than one 
core subject (39 percent compared with 46.8 percent). 

• Combining these two indicators, 58.5 percent of SSC enrollees were on track 
to graduate in four years compared with 48.5 percent of their counterparts 
who attended a different type of school — a 10 percentage point difference. 

s As explained in Appendix A, to estimate the effects of enrolling in an SSC, the estimated effects of win- 
ning an SSC lottery (see Appendix B) are adjusted to account for the proportion of SSC lottery winners who do 
not enroll in an SSC and the proportion of control group members who do enroll in an SSC, using a well- 
known statistical approach called instrumental variables analysis. 

6 While the schools attended by non-SSC enrollees were significantly larger, some of those larger schools 
(for approximately one-eighth of those students) had structures such as small learning communities in place to 
increase the level of personalization. 



ES-5 




New York City Small Schools of Choice 
Table ES.l 

Estimated Effects of SSC Enrollment in Years 1 to 4 of High School 





Target 


Control 




Effect Size 


P-Value for 




SSC 


Group 


Estimated 


(Standard 


Estimated 


Outcome 


Enrollees 


Counterparts 


Effect 


Deviation) 


Effect 


Year 1 of high school (cohorts 1 to 4) 












9th-grade on-track indicator 2 (%) 


58.5 


48.5 


10.0 ** 




0.000 


Earned 10 or more credits 


73.1 


62.3 


10.8 ** 




0.000 


Failed more than 1 semester of a core subject 


39.0 


46.8 


-7.8 ** 




0.000 


Total credits earned toward graduation 15 


11.3 


10.4 


0 9 ** 


0.21 ** 


0.000 


Total number of student observations = 29,81 1 












Year 2 of hieh school (cohorts 1 to 3) 












Earned 20 or more credits (%) 


69.4 


58.3 


11.1 ** 




0.000 


Total credits earned toward graduation 15 


22.3 


19.8 


2.6 ** 


0.31 ** 


0.000 


Total number of student observations = 21,822 












Year 3 of hisih school (cohorts 1 and 21 

Earned 30 or more credits (%) 


69.5 


62.4 


7.1 ** 




0.000 


Total credits earned toward graduation 15 


32.2 


29.7 


2.4 ** 


0.23 ** 


0.000 


Total number of student observations = 13,297 












Year 4 of hiah school (cohort 11 

Graduated from high school 


68.7 


61.9 


6.8 * 




0.013 


Local diploma granted 


24.6 


21.9 


2.8 




0.261 


Regents diploma granted 


39.5 


34.6 


4.9 




0.074 


Advanced Regents diploma granted 


4.4 


5.5 


- 1.1 




0.366 


Total number of student observations = 5,363 













SOURCES: MDRC's calculations use High School Application Processing System data from eighth-graders in 
2004-2005 to 2007-2008, as well as data from New York City Department of Education attendance, course 
credits, Regents exam, transactional, and enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: This table presents the estimated effects for students who have follow-up course credits data. Appendix 
A describes how values in the column labeled "Target SSC Enrollees" are estimated. Appendix A also describes 
how values in the column labeled "Estimated Effect" are estimated. Values in the column labeled "Control Group 
Counterparts" are differences between corresponding values in the first and third columns. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated as: ** = 1 
percent; * = 5 percent. 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 2005, 2006, 
2007, and 2008, respectively. 

a The on-track composite measure indicates whether students earned at least 10 credits in their first year of high 
school and had no more than one semester of failure in a core subject in that school year (English, math, science, 
and social studies). 

b The "total credits earned toward graduation" measure is the aggregate number of course credits earned toward 
fillfilling the New York State graduation requirements. The credit requirements are as follows: 31 core subject 
credits, including 8 credits each of English and social studies; 6 credits each of math and science; 2 credits of arts; 
1 credit of health; and 13 additional credits, including 4 credits of physical education, 2 credits of a foreign 
language, and 7 credits of electives. 
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• During the first year of high school, SSC enrollees earn almost one full credit 
more (0.9 credit) toward graduation than do their control group counterparts. 

These positive effects on the transition into high school during ninth grade were seen 
among nearly all subgroups as defined by students’ academic proficiency, socioeconomic 
status, race/ethnicity, and gender. The effects of SSCs for the second year of high school (using 
data from the first three cohorts) are also positive: 

• Among second-year SSC enrollees, 69.4 percent had earned 20 or more cred- 
its toward graduation as opposed to 58.3 percent of their counterparts in non- 
SSC schools — an 1 1. 1 percentage point difference. 

• Second-year SSC enrollees had accumulated an average of 22.3 credits to- 
ward graduation as opposed to 19.8 credits for their non-SSC counterparts, 
for a difference of 2.6 credits. 

• SSCs continue to increase students’ engagement during their second year of 
high school, as evidenced by the increase in the percentage of students who 
attend school regularly — that is, at least 90 percent of the time — by 6.2 
percentage points (49.0 percent for the non-SSC group compared with 55.2 
percent for SSC enrollees). 

In the third year of high school, positive effects continue to accumulate (according to 
data from the first two cohorts): 

• SSCs increase the percentage of students earning 30 or more credits by 7. 1 
percentage points (69.5 percent for SSC enrollees compared with 62.4 per- 
cent for the non-SSC group). 

• SSCs increase the average number of credits earned toward graduation by 
2.4 credits (32.2 credits compared with 29.7 credits). 

• SSCs increase average attendance during students’ third year of high school 
by 3.0 percentage points and increase the percentage of students who attend 
regularly by 8. 1 percentage points. 

In summary, SSCs consistently improve student academic outcomes during the first 
three years of high school. The next logical question is: To what extent do these academic gains 
translate into increased rates of high school graduation? 
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Effects on Graduation Rates 

For the first cohort of students (the only cohort for whom there are four years of follow- 
up data), the evidence indicates that SSC improvements in students’ academic progress and 
school engagement during the early years of high school translate into higher rates of on-time 
graduation after four years: 

• SSCs increase overall graduation rates by 6.8 percentage points, from 61.9 
percent for students who attend schools other than SSCs to 68.7 percent for 
SSC enrollees. 

• A majority of the SSC effect on graduation rates reflects an increase in re- 
ceipt of New York State Regents diplomas. 7 For this type of diploma, stu- 
dents must pass a series of Regents examinations with a score of 65 points or 
above and pass all of their required courses. 

• SSCs increase the proportion of students (by 5.3 percentage points) who 
passed the English Regents with a score of 75 points or higher, the threshold 
for exempting incoming students at the City University of New York from 
remedial courses. They did not have an effect on math Regents exams. 



What Are the Implications of These Findings? 

Uiese findings speak to the nation’s current focus on high school reform. Much of the 
national discussion focuses on three areas where the education community has struggled to 
demonstrate success: (1) improving the academic outcomes of the most disadvantaged students, 
particularly with respect to high school graduation and college readiness; (2) identifying turn- 
around strategies for historically underperforming schools; and (3) implementing effective 
interventions at scale. This study sits at the nexus of all three themes, and its findings demon- 
strate that, in a relatively short period of time, an effective model can be implemented at scale 
and can improve the academic trajectories of large numbers of traditionally underserved students. 



7 Although the estimated effect of SSCs on the overall high school graduation rate is statistically signifi- 
cant, estimates of SSC effects on graduation rates by type of diploma (p = 0.07) miss the standard of statistical 
significance established for this study (p = 0.05). Thus, comparisons of effects across diploma types are 
suggestive only. Regents exams are administered to all public high school students in New York State. 
Students must pass at least five tests in specified subject areas in order to graduate with a diploma that is 
recognized by the New Y ork State Board of Regents, which sets standards and regulations for all public 
schools. 
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The effects of small high schools of choice described in this report should be under- 
stood through three important lenses: their scale, the particular package of reforms they 
represent, and the group of highly disadvantaged students for whom they occurred. 

Effecting Change at Scale. At capacity, the 105 SSCs in the study sample will serve 
over 45,000 students. That is roughly equivalent to the entire high school population of 
Houston, which is the seventh largest school district in the country. Readers should understand 
the magnitude of the present report’s findings in that context — imagine, ybr a school district the 
size of Houston, increasing the percentage of ninth-graders who are eligible for on-time promo- 
tion by 10.8 percentage points, the percentage of black males in ninth grade who are on track to 
graduate by 8.5 percentage points, or the percentage of high school graduates by 6.8 percentage 
points. Given the scale of the SSC initiative, even seemingly minor gains can be understood as 
affecting thousands of high school students. In fact, the 6.8 percentage point increase in four-year 
graduation rates is roughly equivalent in size to one-third of New York City’s gap in graduation 
rates between white students and students of color. Additionally, because the reported effects of 
SSCs are not the product of a small, targeted intervention but rather of a large system of small 
schools, the effects can be understood as reflecting the mean performance of a model imple- 
mented at scale. Reported effects are not the product of the best or most popular of the SSCs, but 
of 105 schools on average. In other words, the findings represent a real-world test of an interven- 
tion launched at the scale of a large-sized urban school district. 

The SSC Package of Reforms. Students enrolled in SSCs did not just attend schools 
that were small. SSC enrollees attended schools that were purposefully organized around 
smaller, personalized units of adults and students, where students had a better chance of being 
known and noticed, and where teachers knew enough about their charges to provide appropriate 
academic and socioemotional supports. SSCs were not only new but were mission-driven. Their 
recent establishment via a demanding authorization process, which rejected more school 
proposals than it approved, required that a prospective school leadership team articulate an 
educational philosophy and demonstrate how it would motivate teachers, community members, 
and partner organizations around it. And the district’s commitment to acting as a steward for 
new schools throughout the start-up period generated a set of supports and protections as these 
schools got up and running. Finally, SSCs benefited from an influx of external ideas, talent, and 
resources. 

Serving Disadvantaged Students. SSCs were intended to be a viable and accessible 
option for the district’s most disadvantaged students, and over the course of the study period, 
they served a population that almost exclusively comprised low-income students of color. The 
fact that SSCs targeted and served this population gives the reported findings even greater 
policy significance, as it is precisely economically disadvantaged students of color who find 
themselves at the bottom end of the nation’s persistent achievement gap, and who are least 
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likely to graduate from high school on time, if at all. Furthermore, the robust positive SSC 
effects for many different types of students, including young men of color, hold out great hope 
for educational policymakers, practitioners, and researchers who wish to effect change, by 
demonstrating that it is possible to transform a large number of high schools in ways that benefit 
many disadvantaged students. 



Interpreting and Using the Findings 

The reforms implemented in New York City should be considered as a package of inte- 
grated, reinforcing strategies. The effects are not simply the result of closing low-perfonning 
schools or of creating SSCs, but rather a purposeful marriage of the two strategies supported by 
the implementation of several enabling refonns. Decision-makers interested in replicating the 
district’s strategy should devote as much attention to how these refonns were operationalized as 
they do to what was conceptualized. Closing the failing schools would likely not have been 
singularly effective without the intentional creation of a range of viable alternative options to 
educate the displaced students. Similarly, the creation of new schools would likely not have 
gained the traction it did without the introduction of a districtwide choice process that motivated 
previously underserved students and their families to explore their high school options and 
exercise choice. Thus, while this study provides compelling evidence in support of a particular 
small school model, that model cannot be understood as existing in isolation but rather as one 
integral component of a comprehensive and coordinated set of district refonns. 

While these results are unifonnly encouraging, they are still early. Only one of the co- 
horts has been followed through four years of high school up to graduation. The full effects of 
the high school refonn initiative in New York City will not begin to be revealed until the 
remaining three cohorts of students graduate from high school and venture into postsecondary 
education and the labor market. 
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Chapter 1 

Introduction 



Over the last decade, New York City has been the site of a systemwide high school 
reform effort that is unprecedented in its scope and pace. Since 2002, the school district has 
closed more than 20 failing high schools, opened hundreds of new secondary schools, and 
implemented a centralized high school admissions process that serves approximately 80,000 
students per year. 

At the heart of these reforms lie the new schools that in this report are called “small 
schools of choice” (SSCs) — small, nonselective, public high schools serving students in 
grades 9 through 12 that were opened between 2002 and 2008. Serving approximately 100 
students per grade and open to students at all levels of academic achievement, the SSCs in this 
study were created to serve the district’s most disadvantaged students. Prior to the 2002-2003 
school year, New York City’s most disadvantaged public school students had little option but 
to enroll in one of the city’s large, zoned high schools when they made the transition from 
eighth to ninth grade. Many of those schools were low-performing, with graduation rates 
below 50 percent. 

Supported by the Bill & Melinda Gates Foundation, this report presents very encourag- 
ing findings from an unusually large and rigorous study of the effects of SSCs on students’ 
academic achievement in high school. The study benefits from two aspects of the district’s 
systemwide strategy: (1) the unprecedented scale at which SSCs were created, with 123 
opening in the six-year period noted above; and (2) the district’s simultaneous introduction of a 
universal high school choice process, which created lotteries for each SSC that was oversub- 
scribed — that is, for schools that did not have enough spaces for all the students who wished 
to attend them. 

The analysis presented in this report uses data from the high school admissions process 
to identify a sample of students who chose SSCs, but who — because their chosen SSC had 
more applicants than seats available — were assigned via lottery either to that school or to a 
subsequent choice on their list. 1 The analysis includes four annual cohorts of students who 
entered high school in the fall of 2005, 2006, 2007, and 2008 — a total of 21,085 students who 
applied to 105 oversubscribed SSCs. 

The report presents estimates of the effects on students’ academic progress of enrolling 
in a small school of choice relative to what their progress would have been if they had enrolled 

'in rare instances, students are not assigned to any of the choices they list. 
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in one of the wide range of other available public high schools in New York City. 2 For the 
students who constitute the study sample, those other schools were, on average, older and 
larger. 3 To obtain these estimates, school records data are used to compare the academic 
progress of students who enroll in SSCs with that of their control-group counterparts who enroll 
elsewhere. This analysis reveals robust positive effects on students’ academic transition into 
high school, their subsequent progress toward graduation, and their attainment of a high school 
diploma within four years. 

The remainder of this chapter positions these findings by: 

• Describing the unique set of implementation conditions that gave rise to 
SSCs 

• Discussing the ways in which this report’s analysis of SSCs contributes to 
the evidence base around the efficacy of small schools 

• Presenting a research framework that outlines what was studied, what com- 
parisons were drawn, and how broadly the findings can be generalized 



Only in New York 

The New York City public school system is the largest in the United States, with over 
1.1 million students enrolled in more than 1,600 schools. 4 Over the past decade, it has been the 
site of an ambitious effort to reform the high school system, of which the creation of SSCs was 
a central part. 

In 2002, newly elected Mayor Michael Bloomberg identified school reform as a priority 
of his administration, successfully petitioning the New York State legislature to establish 
mayoral control of the school district, 5 and appointing Joel Klein — a nationally recognized 
antitrust lawyer — as its chancellor. Beginning in 2002, the New York City Department of 
Education (DOE) rapidly introduced an ambitious set of reforms organized around the prin- 
ciples of “Leadership, Empowerment, and Accountability,” including: 



2 As explained in Appendix A, to estimate the effects of enrolling in an SSC, the estimated effects of win- 
ning an SSC lottery are adjusted to account for the proportion of SSC lottery winners who do not enroll in an 
SSC and the proportion of control group members who do enroll in an SSC, using a well-known statistical 
approach called instrumental variables analysis. 

3 It is important to remember that the SSCs are not being compared with the large, failing schools they 
replaced. 

4 NYC Department ofEducation (2010a). 

5 Prior to 2003, district governance responsibilities were divided among the mayor, an appointed Board of 
Education, and 32 locally elected school boards. 



2 




• The founding of a training institute for school principals, known as the NYC 
Leadership Academy, and the expansion of an analogous program for teach- 
ers, known as the NYC Teaching Fellows, intended to both attract and train 
nontraditional educators to serve in DOE schools 6 

• An amendment to the collective bargaining agreement, which eliminated the 
prevailing hiring policy that compelled principals to hire teachers with se- 
niority in favor of a policy that gave them more discretion 7 

• The conversion from a management system of regional offices responsible 
for school governance, curricular mandates, and budget allocation to one in 
which schools had greater control over their budgets and educational pro- 
grams, and were responsible for contracting with their choice of “School 
Support Organization” to purchase core services 8 

Together, these systemwide changes, which provide the policy backdrop for the high 
school reforms described below, signaled a shift in responsibility for decisions about budget, 
staffing, and instruction away from district offices to the schools themselves. 

In an effort to address its persistently low four-year graduation rate, which had hovered 
around 50 percent for more than a decade, 9 the district implemented a particularly ambitious set 
of changes at the high school level. The impetus for and scope of those reforms are detailed at 
length in a companion report that identifies two key dimensions of the district’s efforts: (1) an 
overhaul of the stock of existing high schools through the closure of some high schools and the 
creation of others, and (2) the implementation of a centralized high school choice process for all 
rising ninth-graders. 10 

While the changes had systemwide implications, they were specifically intended to 
benefit the district’s most academically and socioeconomically disadvantaged students, who, as 
noted earlier, had historically been served by a limited and often low-performing set of high 
schools. 



6 By the 2008-2009 school year, graduates of the NYC Leadership Academy represented 13 percent of 
New Y ork City public school principals, and graduates of the NYC Teaching Fellows represented 1 1 percent 
of teachers. See NYC Leadership Academy (n.d.) and NYC Teaching Fellows (n.d.). 

7 Daly, Keeling, Grainger, and Grundies (2008). 

beginning in 2007-2008, principals were able to choose their school’s School Support Organizations. 
These organizations provide many of the same services and supports that were historically provided by the 
New York City Department of Education through its regional offices. 

’Between 1992 and 2002, New York City Department of Education graduation rates ranged from 48-51 
percent. See NYC Department of Education (2010b). 

10 Quint, Smith, Unterman, and Moedano (2010). 
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A Changed Set of High School Options 

Be g innin g in 2002, the DOE accelerated and expanded efforts that had been under way 
in the city since the mid-1990s to close low-performing schools and open new small schools in 
their stead. 11 Die chancellor’s reform efforts were supported by a consortium of funders led by 
the Bill & Melinda Gates Foundation — which ultimately invested over $150 million in New 
York City 12 — and were implemented in partnership with the teachers and principals unions. 13 
The resulting changes in the high school landscape transpired with unprecedented scale and 
rapidity. By 2008, 23 dysfunctional high schools — defined as those with graduation rates below 
45 percent — had been targeted for closure, 14 and 216 new small schools had been opened. 

Box 1 . 1 describes the range of new small school types that were opened during this pe- 
riod. While the district advocated a “portfolio” approach and established a variety of models 
ranging from transfer schools designed to serve students who had struggled in conventional 
high schools to specialized schools intended to serve the district’s highest-performing students, 
the predominant model was the small school of choice (SSC): 15 small, nonselective general high 
schools intended to serve grades 9 to 12 at capacity. As conceived, all of the district’s new small 
schools were intended to provide a range of geographically accessible options for students 
whose neighborhood high schools were closing. SSCs, which did not impose academic admis- 
sions requirements, represented a particularly viable option for those students. SSCs gave 
preference to students who (1) had geographic priority (usually residing within the same 
borough as the school), and (2) had attended a school’s open house or the school’s booth at a 
school fair, 16 or who were otherwise “known” to the school. 

Table 1.1 shows the distribution, by school type, of new small schools and their enrol- 
lees between the 2005-2006 and 2008-2009 academic years. The table not only illustrates the 



1 1 A history of small schools in New York City appears in Quint, Smith, Unterman, and Moedano (2010), 
including the efforts undertaken by New Visions for Aiblic Schools — which launched the New Century High 
Schools initiative — that immediately preceded and served as the model for the school creation efforts under 
the Bloomberg/Klein administration. Similarly, the report notes that of the 23 schools closed between 2002 and 
2008, three had been identified prior to that time period. 

12 The Gates Foundation supported the DOE’s new school creation efforts in partnership with the Carnegie 
Corporation of New Y ork and the Open Society Institute, and other systemwide initiatives benefited from more 
than $230 million worth of funding from philanthropies including the Wallace Foundation, the Michael & 
Susan Dell Foundation, and the Eli and Edythe Broad Foundation. See Quint, Smith, Unterman, and Moedano 
(2010) and NYC Department of Education Fund for Public Schools (n.d.). 

1 “’New Visions for Aiblic Schools (2005). 

14 Schools targeted for closure were typically phased out by ceasing admission of first-time freshmen but al- 
lowing enrolled students to advance through the upper grades. See Quint, Smith, Unterman, and Moedano (2010). 

The “of choice” in “small schools of choice,” a term coined by the researchers, is meant to emphasize 
the fact that these nonselective schools are accessible to students of all academic levels. 

16 New York City holds several city and boroughwide school fairs to give students an opportunity to talk 
with representatives of the city’s public schools and learn about their programs. 
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New York City Small Schools of Choice 



Box 1 .1 

Types of New Small Schools Opened Between 2002 and 2008 

General high schools offer a standard core curriculum in addition to elective courses and 
serve students at various levels of academic ability in grades 9-12. 

• Small schools of choice (SSCs) are both small and academically nonselective (123 
opened by the 2008-2009 school year). 

• Other general high schools are small and academically selective (38 opened by the 
2008-2009 school year). 

Transfer schools are small, personalized, full-time schools designed to help overage and 
undercredited students overcome obstacles to graduation (21 opened by the 2008-2009 
school year). 

Middle/high schools, which typically serve grades 6-12 or 7-12, are intended to support 
students’ transition from middle to high school by enabling them to maintain relationships 
with familiar staff members and stay within familiar surroundings (33 opened by the 2008- 
2009 school year). 

Specialized high schools serve students who are high-performing academically and/or 
artistically. Admission usually depends on a student’s score on the Specialized High 
Schools Admissions Test (SHSAT), which is taken during a student’s eighth-grade year (1 
opened by the 2008-2009 school year). 



SOURCE: Quint, Smith, Unterman, and Moedano (2010). MDRC calculations use High School 
Application Processing System data from 2004-2005 to 2007-2008 and school-level administrative 
records provided by the DOE for the 2002-2003 to 2008-2009 school years. 



pace and magnitude of the district’s school creation efforts (the 216 new small secondary 
schools that had opened by 2008-2009 served 17,682 students), but also demonstrates the 
prevalence of the nonselective SSCs, which by 2008-2009 numbered 123 and served 12,448 
first-time ninth-graders (70 percent of those served by all new small schools). 

The SSCs were not just small and nonselective; they also shared four essential features 
with the other new small schools that were founded during this period: 

1 . They were predominantly located in disadvantaged communities whose 
neighborhood high schools were closing. 
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New York City Small Schools of Choice 
Table 1.1 

Number of New Small Schools and Their First-Time Ninth-Grade Student Enrollment, by School Type 







Number of Schools 




Number of First-Time 9th-Graders Enrolled 


School Type 


2005-2006 


2006-2007 


2007-2008 


2008-2009 


2005-2006 


2006-2007 


2007-2008 


2008-2009 


NYC Deoartment of Education 
tnOFd New Small Schools 


















Transfer 2 


8 


9 


14 


21 


- 


- 


- 


- 


Middle/high schools 


17 


21 


27 


33 


1,382 


1,760 


2,314 


2,570 


High schools 


102 


117 


137 


162 


10,666 


12,307 


13,323 


15,112 


Specialized high schools 


0 


1 


1 


1 


0 


65 


37 


90 


General high schools 


102 


116 


136 


161 


10,666 


12,242 


13,286 


15,022 


SSCs 


85 


96 


110 


123 


8,869 


10,219 


11,347 


12,448 


Total number of schools or students 


127 


147 


178 


216 


12,048 


14,067 


15,637 


17,682 



SOURCES: MDRC calculations use High School Application Processing System data from eighth-graders in 2004-2005 to 2007-2008, data 
from New York City Department of Education (DOE) enrollment fdes for the 2005-2006 to 2008-2009 school years, and school-level 
administrative records provided by the DOE for the 2002-2003 to 2008-2009 school years. 



NOTES: Rounding may cause discrepancies in sums and differences. 

Previous year's enrollment files were used to determine whether or not a student was a first-time ninth-grader. 

“Transfer schools serve students who have been previously enrolled in a traditional high school, and are thus not intended to serve first-time 
ninth-grade students. Enrollment numbers are therefore not provided for this school type. 




2. They were authorized through a demanding competitive proposal 
process that emphasized the common design principles of academic 
rigor, personalization, and community partnerships. 

3. They benefited from an infusion of outside resources: new principals and 
teachers, partnerships with intennediary organizations, and start-up fund- 
ing from the district and its philanthropic partners. 

4. They received policy protections during their start-up period. 

A Focus on Historically Underserved Communities 

During the period under study, the closure of underperforming schools and the opening 
of new small schools were concentrated in the city’s poorest boroughs — Brooklyn and the 
Bronx. Nineteen of the 23 large high schools that were closed by 2008 and 128 of the 216 new 
small secondary schools that opened were located in those two boroughs. Figure 1.1 illustrates 
the student population shift in Brooklyn and the Bronx as of the 2007-2008 school year at the 
19 closed high schools and the 128 new small schools that were open at that point. 17 By 2007, 
those 128 schools served 38,922 students across grades 9-12 (with the 85 SSCs among them 
serving 28,016). The number of students served by the new small schools actually exceeds the 
36,892 students whom the large closing schools had served six years earlier. 

A Rigorous Planning Process and Common Design Principles 

As noted above, the new small schools were authorized through a competitive proposal 
process that emphasized three core elements: 18 

• Academic Rigor. Schools were expected to set high expectations for stu- 
dents and to offer a standards-based curriculum aligned with New York State 
graduation requirements. In fact, schools were encouraged to develop col- 
lege-ready standards that exceeded basic graduation requirements and em- 
phasized higher-order skills such as critical thinking. 



17 Figure 1.1 provides whole school enrollment information (grades 9-12) for the 2002-2003 school year 
through the 2007-2008 school year. In order to provide data for the years prior to the period of the present 
study, the authors rely on New York State school-level data, which were only available through the 2007-2008 
school year at the time of publication. Thus, unlike other references in this analysis, the figure does not include 
the 2008-2009 school year. 

ls Quint, Smith, Unterman, and Moedano (2010). 
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New York City Small Schools of Choice 
Figure 1.1 

Student Enrollment in Closing High Schools, Small Schools of Choice, 
and Other New Small Schools in Brooklyn and the Bronx 
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SOURCE: MDRC calculations use High School Application Processing System data from eighth-graders in 
2004-2005 to 2007-2008, as well as New York State Report Card and school-level administrative records 
provided by the New York City Department of Education for school years 2002-2003 through 2007-2008. 

NOTE: The figure presents whole school enrollment numbers corresponding to the following counts of 
schools in Brooklyn and the Bronx: 19 schools that ceased admitting new ninth-grade students between the 
2002-2003 and 2007-2008 school years, 85 small schools of choice, and 43 other new small schools. 



• Personalization. Schools were to be small not only in size but also in func- 
tion. Structures such as “Advisory ,” 19 reduced teacher load, and common 
planning time (in which teachers meet together to discuss their students’ 
progress and problems) were recommended to ensure that all students were 
known well and to promote strong, sustained relationships between students 
and teachers. 

• Community Partnerships. The majority of new small schools were theme- 
based, with their curriculum organized around a theme such as business or 
law. Through partnerships with business and community partners, schools 
were intended to offer learning opportunities outside the classroom and to in- 
fuse classroom instruction with relevant real-world examples. Partners were 
expected to bolster school capacity in areas ranging from curriculum and in- 
struction to youth development and community outreach. 

An Infusion of External Resources 

The new small schools were started from scratch, fueled by new ideas, talent, and capi- 
tal that came from sources beyond the school district. 

• The new small schools were founded by teams of teachers and administrators 
that had self-affiliated and participated in a rigorous planning and proposal 
process in order to win school approval. These planning teams developed the 
mission and vision for each new school as well as the planned curriculum 
and student services. 

• Teams were encouraged to involve community partners, and the majority did 
so. While these partners were sometimes small community-based organiza- 
tions, they were more likely to be established education intermediaries — 
nonprofit organizations that served both as fiscal agents for distributing grant 
funds to schools and as central sources of experience and technical support 
related to the creation and operation of small schools. Most of these interme- 
diary organizations received funding from the Bill & Melinda Gates Founda- 
tion to start schools in New York City, and the large majority had started new 
schools before receiving Gates funding to do so. Once the new small schools 



19 “Advisory” (also known as “Family Group”) is a counseling model whereby teachers, administrators, 
and other adults in the building act as “advisors” to small groups of students, with whom they meet as part of 
the regular schedule to address academic and socioemotional issues. 
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were open, intermediaries were charged with providing ongoing technical as- 
sistance, largely in the areas of leadership development, instructional support, 
and college-readiness services. 

• Finally, new small schools were provided with start-up grants to support 
costs associated with implementation. Those schools that were affiliated with 
Gates-funded intermediaries received four- or five-year grants averaging 
$400,000 per school. Other new small schools received supplemental grants 
directly from the DOE. 

Support for Start-up 

As noted earlier, the district’s efforts to create new small schools was informed by pre- 
vious local school creation efforts. The implementation challenges that had been experienced 
historically were well documented, and district planners put several measures in place to buffer 
and support this wave of new small schools from similar pitfalls during their nascent years: 

• Schools opened with only one founding grade of students (ninth grade), gradually 
phasing in by admitting an additional cohort each year until reaching capacity for 
all four grades of a standard high school curriculum. 

• New small schools that opened between 2002 and 2007 were not required to serve 
English language learners or special education students in their first two years of 
start-up operations while internal capacity was still being built. 20 

• District offices that were devoted to new school support were intended to facilitate 
procurement and hiring issues (which were difficult logistically because the schools 
were new administrative entities) and to support issues related to school facilities 
and shared campuses. 

In summary, between 2002 and 2008, the district established more than 200 new small 
schools, which were distinguished by several key features beyond their size. Over the course of 
that period, SSCs — academically nonselective small schools serving mostly disadvantaged 
students in grades 9 through 12 — emerged as a prevalent and established school model, 
particularly in those communities that had previously been served by the large zoned high 
schools. 



20 By their third year of operations, schools that opened between 2002 and 2007 were required to admit 
students with special needs. Schools that opened in 2008 or later were expected to serve students with special 
needs from their inception. 
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Choice for All 

Concurrent with the change in the supply of high school options, the DOE overhauled 
the process by which students express demand for those schools. Historically, high school 
admission was managed through an uncoordinated set of processes in which some families were 
able to exercise choice over the high schools attended because they were informed about the 
options that were available to them and because they were able to successfully fulfill admissions 
criteria. However, many students — especially those in disadvantaged communities — were 
enrolled into their neighborhood high school as a matter of course. Then, in the 2003-2004 
academic year (for students slated to enter high school in fall 2004), the city introduced the 
High School Application Processing System (HSAPS), a centralized choice process that was to 
govern the placement of all entering ninth-grade students. 

Box 1.2 details the features that distinguished HSAPS from its antecedent. Notably, the 
new process was intended to compel all rising ninth-graders to exercise choice (by requiring 
that they each select up to 12 high schools) and to render admissions decisions in an objective 
and standardized manner. HSAPS used a computer-based algorithm to match student choices 
against school eligibility criteria in order to produce a single assignment for every participant. 

Figure 1.2 illustrates the channels by which students move from eighth to ninth grade 
since the introduction of HSAPS. Table 1.2 presents HSAPS participation rates over the course 
of the study period. Viewed together, Figure 1.2 and Table 1.2 illustrate several key operational 
outcomes associated with the implementation of HSAPS. 

In terms of efficacy: 

• The process functions at an unprecedented scale, averaging just under 80,000 
participants each year between the 2004-2005 and 2007-2008 school years. 

• HSAPS produced an offer for the vast majority of participants — 91 percent, 
on average — and most students are matched to one of their top three choic- 
es, at a rate that increased from 65.7 percent in 2005 to 78.2 percent in 2008. 

In other words, on average, approximately 72,500 students are assigned to 
high school via HSAPS each year. 

In terms of participation: 

• Among those enrolled in eighth grade, there was near-universal participation, 
with 92 percent of DOE eighth-graders participating on average. 
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New York City Small Schools of Choice 

Box 1.2 

Key Differences Between New York City’s High School Application 
Processing System (HSAPS) and the System It Replaced 

• HSAPS involves all students. In the previous system, participation was voluntary, 
which resulted in a self-selected applicant pool that represented more sophisticated 
students and parents. Under HSAPS, all rising ninth-graders are required to exercise 
choice by submitting an application listing up to 12 high school programs. The city 
launched extensive public outreach efforts to inform students and parents of their op- 
tions, publishing informational materials in languages ranging from Creole to Urdu, 
distributing a phone book-sized high school directory (and an online analog) with in- 
formation about every high school program in the district, and hosting multi-day high 
school fairs where representatives from every high school in the city provide informa- 
tion about their programs to tens of thousands of students and families. 

• HSAPS sets standards for all schools. Previously, despite a nominally “centralized” 
process, schools often had widely varying application requirements and deadlines, 
and rendered admissions decisions independently. As part of the new system, an Of- 
fice of Student Enrollment was established to standardize the steps of the admissions 
process across many different types of schools, and to funnel all applications and ad- 
missions decisions through the centralized HSAPS system. 

• HSAPS guarantees all students a single “offer.” Under the previous system, 
schools were independent arbiters of their admissions. As a result, some participants 
(often high-performing students) received offers from multiple schools while others 
received none. Other unaware or unmotivated students didn’t participate at all and 
were often siphoned into zoned, neighborhood schools. HSAPS was designed to pro- 
vide every student with a single offer of admission that, barring extenuating circum- 
stances, was intended to be his or her fin a l placement. 

SOURCES: Abdulkadirouglu, Pathak, and Roth (2009); Hemphill and Nauer (2009). 



• Among those who enter ninth grade, the vast majority do so via HSAPS — 
on average, 86 percent of all first-time students (including those who arrive 
new to the system and may not have had the opportunity to participate) and 
97 percent of those first-time students advancing from DOE middle schools. 

Together, these statistics demonstrate that the DOE was able to successfully implement a 
districtwide choice process that operated at scale and achieved its primary objective of producing 
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New York City Small Schools of Choice 
Figure 1.2 

Student Flow from Eighth-Grade Choice to Ninth-Grade Enrollment 
Among Students Enrolled in DOE Schools 



Choice Enrollment 

Among Eighth-Graders Among First-time Ninth-Graders 

Enrolled in DOE Schools Enrolling in DOE Schools 




NOTES: DOE = New York City Department of Education. HSAPS = High School Admissions Processing System. 

a A student may move into the district or transfer from a parochial school, for example, after the HSAPS process has concluded, 
or may not receive a match via HSAPS. In those instances, a student enrolls through a borough enrollment office. 









New York City Small Schools of Choice 
Table 1.2 

HSAPS Participation Patterns by Year 



Students 


2005 


2006 


2007 


2008 


General DarticiDation 










Total number of HSAPS participants (including students 
not enrolled in DOE schools) 


81,643 


82,464 


77,999 


75,577 


Percent of all 8th-grade participants receiving a match 


89.9 


89.2 


93.0 


93.2 


Percent of all 8th-grade participants matched to choice 1-3 


65.7 


71.4 


73.8 


78.2 


ParticiDation among students enrolled in DOE schools 










At the point of 8th-grade choice (spring) 










Total number of 8th-graders 


84,774 


83,122 


80,912 


78,984 


Percent of all 8th-graders who participate 


91.8 


92.9 


92.8 


92.0 


At the point of 9th-grade enrollment (fall) 










Total number of first-time 9th-graders 


84,191 


81,945 


80,212 


78,209 


Percent of all first-time 9th-graders who participated 


85.1 


86.3 


86.0 


85.5 


Total number of first-time 9th-graders advancing from DOE schools 


72,311 


70,728 


69,580 


67,417 


Percent of all first-time 9th-graders advancing from DOE schools 
who participated 


96.2 


97.2 


96.7 


96.9 



SOURCES: MDRC's calculations use High School Application Processing System (HSAPS) data from eighth- 
graders in 2004-2005 to 2007-2008, as well as data from New York City Department of Education (DOE) 
enrollment files from the 2005-2006 to 2008-2009 school years. 



a high school placement for nearly every student. By 2008, the last year of the study period, 
HSAPS processed applications for 75,577 students and matched nearly 80 percent of them to one 
of their top three choices. No longer was “choice” the exclusive territory of informed parents and 
students; it was now the modus operandi for almost all rising ninth-graders in New York City. 



Contributing to the Evidence Base Around Small Schools 

Among many refonns that have been suggested to improve secondary education, the 
notion of small schools has held a singular appeal to school districts, policymakers, and phi- 
lanthropists. Reports estimate that nearly every major American urban district, and all but four 
states in the country, have undertaken efforts to create new small schools or to transform large 
schools into campuses of “small learning communities” (SLCs) — cohorts of students who take 
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their core classes together with the same group of teachers. 21 Spurred by an infusion of re- 
sources from private funders and government sources, the movement to make high schools 
smaller has gained traction at the federal, state, and local levels. 

The national movement to create small schools had grassroots beginnings as early as 
the 1960s, when urban educators and community organizations began implementing smaller 
school structures as an alternative to the large high schools that dominated the landscape, many 
of which were failing. Notable among early small school initiatives were two based in New 
York City that served as precursors to the SSCs in this analysis: (1) the school creation work in 
East Harlem’s District 4 in the late- 1970s and 1980s, where the introduction of several small 
secondary schools (including Deborah Meier’s Central Park East Secondary School) as “choice- 
based” alternatives to the neighborhood zoned schools was credited with improved student 
outcomes among the district’s largely at-risk student body; and (2) the Annenberg-sponsored 
work during the 1990s, through which campuses of wall-to-wall small schools were created to 
replace large, comprehensive high schools. 

By the late 1990s, small schools had emerged as a national reform strategy championed 
by affinity groups (such as the Coalition of Essential Schools and the Small Schools Workshop) 
and professional organizations (such as the National Association of Secondary School Princi- 
pals), and proliferated through district and foundation-led initiatives in several major cities — 
including Chicago, Philadelphia, Boston, and Oakland in addition to New York. In 2000, the 
Bill & Melinda Gates Foundation launched a national campaign to improve failing urban high 
schools, with small schools as its centerpiece. 22 Within four years, the Gates Foundation had 
granted $735 million toward small school creation (or conversion) efforts at 1,500 high 
schools. 23 Thus, by the first decade of the twenty-first century, the small school approach that 
had begun as a “quiet revolution” 24 in mostly poor, minority communities was being pursued as 
a scalable reform strategy by the largest education reform funder in recent American history. 

Despite small schools’ growing popularity and widespread implementation, there is a 
dearth of reliable and consistent information about their effectiveness. Much of the literature 
about small schools converges around a common theory of change whereby smaller school size 
promotes stronger and more multidimensional relationships between students and adults. These 

2 'Many large schools across the country have been restructured into SLCs as a means of fostering a more 
personalized environment and stronger bonds between students and their teachers. By 2008, 46 of 50 states and 
the District of Columbia had received multimillion-dollar federal SLC grants. See U.S. Department of 
Education (n.d.). 

22 While the Gates Foundation’s stated goal was to support the creation of schools that possessed seven 
“Attributes of High Performing Schools,” it was the structural focus on school size (not to exceed 100 students 
per grade) that became the foundation’s gj'ant-making hallmark. See Evan et al. (2006). 

23 Miner (2005). 

24 Fine (2005). 
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enhanced relationships, in turn, produce increased levels of student engagement and better 
position teachers to identify and respond to students’ academic and social needs. 25 

Research on small schools suggests that they can produce effects on key outcomes, in- 
cluding higher levels of student achievement and lower dropout rates. 26 Some literature suggests 
that these effects are most pronounced among disadvantaged students. 27 While a full review of 
the literature is beyond the scope of this report, the voluminous amount of research on the 
subject of small schools is a testament to the educational community’s interest in the model. 28 
Unfortunately, given the nonexperimental nature of this research, the studies cannot establish a 
causal link between small schools and impacts on student outcomes. 



Studying New York City’s Small Schools of Choice 

The unique set of conditions that have existed in New York City over the past decade 
provide researchers with the opportunity to launch a study that overcomes two limitations of 
prior work: 

• The analysis takes advantage of the lotteries created within the high school 
admissions process to identify a valid comparison for measuring the effects 
of SSC enrollment on student achievement. 

• The scale of the district’s reforms provided an unusually large sample — 105 
schools and more than 20,000 students — constituting a representative sam- 
ple of a large-scale district reform. 

The primary research question of the present study is: What effects do small schools of 
choice have on students’ transition into high school, their progress toward graduation, and their 
ability to graduate from high school compared with what these outcomes would have been had 
students enrolled in high school elsewhere? 

To answer this question, it is important to restate the basic premise of the research de- 
sign: In instances where SSCs have more available applicants than seats, HSAPS creates school 
lotteries to render admission decisions. 29 Researchers compare the academic progress of 



25 See Finn and Voelkl (1993); Lee and Loeb (2000); Wasley et al. (2000); Klem and Connell (2004). 

26 For student achievement, see Flaller, Monk, and Tien (1993); Flowley (1989); Flowley and Fluang 
(1991); Lee and Smith (1997). For dropout rates, see Pittman and Flaughwout (1987); McMullen, Sipe, and 
Wolf (1994). 

27 See Lee and Smith (1993); Lee and Smith (1995); Lee and Smith (1997). 

28 See the Bibliography at the end of this report for selected works on small schools in New York City. 

29 In order to be oversubscribed for lottery purposes, an SSC must have more available applicants (who have 
not already been matched to one of their prior school choices) than seats. Thus, some schools that appear to have 

(continued) 
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students who win these lotteries and attend an SSC with that of their control group counterparts 
who enroll elsewhere. 30 

Recall, too, that SSCs are more than just small, stand-alone institutions serving ninth- 
through twelfth-graders that do not screen students based on their prior academic achievement. 
While structurally similar to the thousands of small high schools that now exist across the 
country, SSCs have the other distinctive features described in detail earlier: 

• They serve a concentration of highly disadvantaged students. 

• They are not only small in overall size but are organized to have smaller, 
more personalized units in which teachers work together and students see a 
smaller number of teachers over a given period of time. 

• They were created through a structured and demanding process that mobi- 
lized teachers, principals, and partner organizations. Many of the teachers 
and principals were new to the system, and the majority of schools were 
founded in partnership with intermediary organizations that had experience 
opening new schools. 

• They received policy protections and a range of supports from the district 
during their start-up phase. 

In addition, SSCs were created amidst an evolving high school reform landscape, which 
has implications for the comparison drawn in this study. By 2008, a student entering high 
school through HSAPS could choose from over 400 schools, representing a wide range of 
options in terms of their sizes, themes, academic programs, and extracurricular offerings. Those 
students who chose SSCs but were randomized elsewhere ultimately enrolled in schools that, 
while generally larger and older than the SSCs, varied along many other dimensions (including 
theme and organizational structure, for example). Thus, this study does not provide a direct 
comparison of small schools to large schools or of the SSCs to the failing schools they replaced. 

Finally, while the natural occurrence of lotteries provides a unique research opportunity, 
it also defines the schools and students included in the study sample as those who listed an SSC 
as one of their choices and participated in a lottery for that choice. Thus: 



been oversubscribed (for example, a school described in the DOE High School Handbook as having had 500 
“total applicants” for 108 “program seats”) may not have actually been oversubscribed for the purposes of 
generating a lottery. See Appendix A for a detailed description of how HSAPS assigns students to SSCs. 

30 As noted earlier and explained in Appendix A, to estimate the effects of enrolling in an SSC, the esti- 
mated effects of winning an SSC lottery are adjusted to account for the proportion of SSC lottery winners who 
do not enroll in an SSC and the proportion of control group members who do enroll in an SSC, using a well- 
known statistical approach called instrumental variables analysis. 
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• Although the majority (85 percent) of SSCs had at least one lottery over the 
course of the study period (in other words, nonlottery schools are the excep- 
tion to the rule), the present study does not attempt to generalize beyond the 
schools included in the sample. 

• Similarly, while the vast majority of SSC enrollees arrive via HSAPS, this 
study does not attempt to generalize its findings to non-HSAPS enrollees. 

The following chapters present the methodology and results of this analysis. Chapter 2 
describes the study design and analysis that was used to generate the present findings. Chapter 3 
compares the study SSCs with the schools that were attended by students in the comparison 
group and presents the estimated effects of enrolling in an SSC on the transition to high school, 
progress toward graduation, and ultimately graduation rates. Chapter 4 reprises the report’s key 
themes and outlines a learning agenda for future work. 
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Chapter 2 

Research Design and Analysis 



As explained in Chapter 1, New York City’s High School Application Processing Sys- 
tem (HSAPS) has been in place since 2004 to assign students from across the district to high 
school. Lotteries for “small schools of choice” (SSCs) are a little-known byproduct of that sys- 
tem. 1 To understand the context for this chapter, recall that, unlike the former system, in which 
better-informed students were most likely to go to the higher-performing schools while less 
savvy students were often relegated to the city’s underperforming schools, HSAPS requires all 
eighth-graders to indicate, in order of preference, up to 12 high schools that they would like to 
attend. When an SSC has more applicants than available seats, a randomized, lottery-like 
process determines which students are assigned to that SSC. 2 

Randomization is useful for evaluating interventions such as SSCs objectively and reli- 
ably, in that it allows researchers to compare groups of study participants (the “sample” or 
“study sample”) that differ only in terms of whether they experience the intervention under 
study (the “treatment group”) or do not experience it (the “control group”). In other words, as a 
group, the study participants start off with the same characteristics, on average — that is, they 
are truly comparable. At the end of the study period, then, any differences between the two 
study groups can be reliably attributed to the intervention. 

As noted in Chapter 1, the HSAPS system and the SSC lotteries allowed researchers to 
launch an unusually large and rigorous study. In this evaluation of New York City’s SSCs, ob- 
served differences in average academic performance and achievement (and other select academ- 
ic outcomes) between students who win SSC lotteries (“SSC lottery winners” in this report) and 
students who lose them (“control group members” in this report) are valid estimates of the ef- 
fects of winning an SSC lottery. 3 



’For the purposes of identifying the study sample, SSCs were defined as high schools that were intended 
to serve grades 9 through 12 rather than grades 6 through 12 or 7 through 12, were founded in 2002 or later, 
and used the “limited unscreened” selection method in F1SAPS. Schools that use the limited unscreened selec- 
tion method do not impose academic requirements but instead give preference to students who live within a 
certain geographic area and have attended a school’s open house or the school’s booth at a school fair, or who 
are otherwise “known” to the school. 

2 Abdulkadiroglu et al. (2009) and Flastings, Kane, and Staiger (2006) also base their research on district- 
wide student assignment processes that create randomized lotteries (in Boston and in Charlotte-Mecklenburg, 
North Carolina, respectively). 

3 The evaluation research literature refers to the type of validity implied here as “internal validity.” 
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However, the presence of the lotteries also presents two methodological challenges. 
The challenges arise because a number of students participate in HSAPS lotteries for more 
than one school on their rank-ordered list of choices. Appendix A describes how these chal- 
lenges are overcome. The two challenges are as follows: 

• One-fourth of the participants in SSC lotteries lost a lottery for a school 
that they ranked higher than the SSC in whose lottery they were partici- 
pating. For example, a student could lose a lottery for his first-choice school 
and then be in a lottery for his second-choice school, which is also an SSC. 

For reasons described in Appendix A, this situation could, in theory, produce 
pre-existing differences between SSC lottery winners and control group 
members. In practice, however, there is no observable difference between the 
two groups. 4 In addition, it is possible to estimate students’ probability of 
winning prior lotteries and thereby control for it statistically. Furthermore, as 
discussed in Chapter 3, the estimated effect of SSCs for the full study is vir- 
tually identical to that for the three- fourths of sample members who were not 
in a prior lottery. 

• HSAPS can legitimately assign a student who loses a lottery for one 
SSC to a different SSC. Thus, losing an SSC lottery does not mean the 
same thing as not being assigned to an SSC. Consequently, the effect of 
winning or losing an SSC lottery is not the same as the effect of being as- 
signed or not being assigned to an SSC. 5 Indeed, the effect of winning an 
SSC lottery does not have a meaningful interpretation. Consequently, all 
findings in the present study are converted into estimates of something that 
does have a meaningful (and policy-relevant) interpretation: the effect of 
enrolling in an SSC. 6 This conversion is based on data for all SSC lottery 
winners and all control group members. Appendix A describes how it is 
made and the assumptions upon which it is based. 

This chapter is organized into the following sections in order to help readers understand 
how the findings that follow in Chapter 3 were obtained: (1) how students and SSCs interact 
with the HSAPS assignment process to create randomized SSC lotteries; (2) how estimates of 

4 If these baseline characteristics were randomized, then unobserved characteristics also should have been 
randomized. 

5 The effect of being assigned to an intervention or treatment is referred to in the literature as the effect of 
“intent to treat.” 

6 The effect of enrolling in an SSC is an example of what is referred to in the literature as a “local average 
treatment effect.” Students are considered to be enrolled in an SSC if they were enrolled at any time during or 
before the follow-up year represented by a given analysis. 
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effects of winning an SSC lottery are obtained from follow-up data for lottery winners and con- 
trol group members; (3) why and how these estimates are converted into estimated effects of 
enrolling in an SSC; (4) the sizes of samples used for these analyses and their data availability; 
(5) the primary sources of data used; and (6) the generalizability of findings obtained. Appendix 
A provides further detail. 



How HSAPS Creates SSC Lotteries 

As noted above, HSAPS assignment is based on rank-ordered preferences submitted by 
all rising ninth-graders for up to 12 high schools. This is how students and their families express 
school choice. HSAPS assignment is also based on indications from each high school of its 
priorities for students. Because SSCs do not screen students academically, their priorities are not 
based on information about students’ past performance. Instead, these priorities are based solely 
on students’ geographic proximity and whether or not they are “known” to the SSC (by having 
contacted it, visited it, or met with one of its representatives). Geographically, most SSCs dis- 
tinguish only between residents of their borough and all other New York City residents. 7 Within 
these categories, the highest priority is given to students who both satisfy the school’s geograph- 
ic preference and are known to the school, whereas the lowest priority is given to students who 
do not carry a geographic preference and are not known to the school. 

HSAPS assigns students to schools based on students’ choices and schools’ priorities. 
The order in which HSAPS chooses each of the roughly 80,000 participants each year for as- 
signment is determined randomly. During the assignment process, as schools begin to fill up, 
their student priorities begin to take effect. For example, if a student’s first choice is already 
filled with students of higher priority (based on their “known” and geographic status), that stu- 
dent cannot be assigned to that school. Instead, he is assigned to the next school on his list that 
has available space. Consider what happens each time HSAPS tries to assign a student to a 
school that is full. If that school gives higher priority to Student A, for example, than to some 
students who are already assigned there, Student A “bumps” the last student assigned who was 
the lowest priority for that school (Student B). Student B is then assigned to his or her next- 
preferred school that has available slots, possibly bumping another student who had been placed 
there previously. This process continues until HSAPS works through the entire randomly or- 
dered list of incoming students. 

A little-known byproduct of the rules of HSAPS assignment is a randomized lottery for 
each SSC that is oversubscribed in a given year. (SSCs that are not oversubscribed in a given 



7 Some SSCs have three geographic priorities: (1) residents of a nearby catchment area, (2) other residents 
of their borough, and (3) other residents of New York City. 
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year do not have a lottery for that year.) This is because the “winners” and “losers” in a given 
lottery are determined solely by the random order in which HSAPS assigns its participants. Stu- 
dents who are assigned early in the process become lottery winners, and students who are as- 
signed later in the process become lottery losers. In this way, eligible and available students are 
randomly assigned either to the SSC (“lottery winners”) or to its control group. A further impor- 
tant feature of each SSC lottery (explained in Appendix A) is that it only involves students from 
within one of the priority categories described earlier (for example, being both known and fulfil- 
ling geographic preference). 8 

In many ways, SSC lotteries are like other lotteries that are used to assign students to 
schools or, more generally, used to allocate scarce resources to individuals or groups. However, 
two features of SSC lotteries distinguish them from most others. First, SSC lotteries are “invisi- 
ble” to the participating students and their parents; they know only that students are assigned to 
one of their chosen schools, but not how that process occurs. Second, a single student can be in 
multiple lotteries. For example, a student could lose a lottery for his first-choice SSC and win a 
lottery for his second-choice SSC (assuming he has indicated more than one SSC in his list of 
12). This student is a control group member for the first SSC and a lottery wimier for the second 
SSC. As noted above, Appendix A demonstrates that multiple lottery participation does not 
pose a threat to the validity of estimated effects of winning an SSC lottery if the analysis proper- 
ly accounts for this phenomenon. 



How Effects of Winning an SSC Lottery Are Estimated 

Students who win an SSC lottery are assigned by HSAPS to that SSC. These students 
are analogous to treatment group members for a medical trial. Students who do not win a par- 
ticular SSC lottery and are assigned to some other public high school in New York City become 
control group members for the SSC whose lottery they lost. Differences in mean future out- 
comes — such as credit accumulation — for lottery winners and control group members, there- 
fore, are valid estimates of the effects of winning an SSC lottery. These estimates are reported in 
Appendix B and are the starting point for the present analysis. 

Randomization is crucial for such estimates because in a large sample, randomization 
creates a “treatment group” (in this case, the SSC lottery winners) and a “control group” (the 
lottery losers) that are the same at baseline in all ways, observable or not, and can therefore be 
compared in a meaningful way. In the current study, therefore, any future differences between 
the two groups must be the result of winning an SSC lottery. Because the sample of students for 

8 A student’s level of preference (or rank order) for an SSC does not affect his priority for it. For example, 
the priority for a student who is in an SSC lottery for his first choice is the same as that for a student who is in 
the same lottery for his twelfth choice. 
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a single lottery is typically small, pre-existing differences between its lottery winners and con- 
trol group members occur by chance. Winners of some lotteries will have stronger average aca- 
demic backgrounds than will control group members, whereas for other lotteries the reverse will 
be true. For some lotteries these differences will be large and for other lotteries these differences 
will be small. Because pre-existing differences occur by chance, their average approaches zero 
(that is, they amount to essentially no differences) when lottery samples are combined because 
the sample size for all lotteries together is much larger than that for any single lottery. 

The full sample for the present study represents 297 SSC lotteries with 12,978 observa- 
tions on lottery winners and 17,981 observations on control group members who were eighth- 
grade HSAPS participants in the 2004-2005 through 2007-2008 school years. Because, as 
noted, some students participate in more than one lottery, the total number of students in the full 
sample (21,085) is less than the total number of student observations (30,959). 

Table 2.1 provides strong evidence of the baseline equivalence of the present sample of 
SSC lottery winners and control group members. This table compares the two groups in terms 
of a number of characteristics that typically predict high school success. 9 Hie first column in the 
table reports means for SSC lottery winners. The second column reports means for control 
group members. The third column reports differences between means for the two groups and 
the fourth column reports the statistical significance (probability value, or p-value) of these dif- 
ferences — or the probability that an observed difference for a sample could have occurred by 
chance if there were no difference in the population. 10 As can be seen, all differences are quite 
small and none is statistically significant. Thus, differences in mean future outcomes for SSC 
lottery winners and then control group counterparts provide valid estimates of the effects of 
winning an SSC lottery. 



Converting Estimated Effects of Winning an SSC Lottery into 
Estimated Effects of Enrolling in an SSC 

The previous section demonstrates that observed differences in average future outcomes 
for SSC lottery wimiers and control group members are valid estimates of the effects of winning 



9 Means for SSC lottery winners are computed directly from data for their pooled sample. Means for con- 
trol group members are computed as weighted averages of means for each SSC lottery, with weights propor- 
tional to the number of winners for each lottery. This accounts for differences among lotteries in their ratio of 
lottery winners to control group members. 

^Conventional practice (although conventions vary) is to consider a difference statistically significant 
(and thus real) if its p-value is 0.05 or less. 
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New York City Small Schools of Choice 
Table 2.1 

Baseline Characteristics of SSC Lottery Participants: 
First Year of High School, Cohorts 1 to 4 





SSC 

Lottery 


Control 

Group 


Estimated 


P -Value for 
Estimated 


Characteristic (%) 


Winners 


Members Difference 


Difference 


Race/ethnicity 


Hispanic 


47.3 


47.9 


-0.6 


0.480 


Black 


43.6 


43.2 


0.4 


0.641 


Other 


7.9 


7.5 


0.3 


0.428 


Male 


46.0 


45.5 


0.5 


0.525 


Eligible for free/reduced-price lunch 


84.0 


84.5 


-0.5 


0.467 


English language learner 


8.4 


7.6 


0.8 


0.114 


Special education 3 


6.6 


6.7 


-0.1 


0.826 


Overage for 8th grade b 


16.7 


18.1 


-1.4 


0.153 


8th-grade reading proficiency 3 


Did not meet standards (level 1) 


6.9 


6.6 


0.3 


0.486 


Partially met standards (level 2) 


60.5 


61.4 


-0.8 


0.328 


Fully met standards (level 3) 


28.4 


27.6 


0.8 


0.287 


Met standards with distinction (level 4) 


0.7 


0.7 


0.1 


0.580 


8th-grade math proficiency 3 


Did not meet standards (level 1) 


18.8 


19.2 


-0.3 


0.628 


Partially met standards (level 2) 


45.1 


44.9 


0.3 


0.759 


Fully met standards (level 3) 


32.8 


31.9 


0.9 


0.238 


Met standards with distinction (level 4) 


2.3 


2.2 


0.1 


0.598 


Total number of student observations = 30,959 



SOURCES: MDRC's calculations use High School Application Processing System and New York 
City Department of Education (DOE) state test data for eighth-graders from the 2004-2005 to 
2007-2008 school years, as well as data from DOE enrollment files from the 2005-2006 to 2008- 
2009 school years. 

NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for 
the difference between SSC lottery winners and control group members are obtained from a 
regression of a given baseline characteristic on a series of indicator variables that identify 
each lottery plus an indicator variable that equals 1 for lottery winners and 0 for lottery losers. The 
coefficient on the latter indicator variable equals the difference in the mean baseline characteristic 
for lottery winners and control group members. The value for control group members equals the 
corresponding value for SSC lottery winners minus the estimated difference between lottery 
winners and control group members. To facilitate computation, all variables are centered on the 
mean value for the lottery they represent. This approach is equivalent to directly accounting for 
each lottery by adding a 0/1 indicator variable for it (Wooldridge, 2000). In some cases, rounding 
may cause slight discrepancies. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

(continued) 
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Table 2.1 (continued) 

A chi-square test was used to assess the statistical significance of the overall difference between 
lottery winners and control group members reflected by the full set of baseline characteristics in 
the table. The resulting chi-square value is not statistically significant (p-value = 0.387). 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 
2005, 2006, 2007, and 2008, respectively. 

“This sample includes special education students who can be taught in the regular classroom 
setting. Special education students classified by the DOE as requiring collaborative team teaching 
services or self-contained classes are not part of the sample. 

b Lottery participants are classified as "overage for eighth grade" if they were 14 or older on 
September 1 of the eighth-grade school year. 

“Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade 
level for state math and reading exams. Due to missing test scores, the sum of levels 1 -4 may not 
add to 100 percent. 

an SSC lottery — an important starting point for this analysis, as noted earlier, and presented in 
Appendix B. However, this is not the whole story. About 7 percent of SSC lottery winners do 
not enroll in an SSC and thus do not experience this type of high school. This is similar to a 
medical trial in which some treatment group members do not take their assigned medicine (that 
is, they do not receive their intended treatment). In such cases, differences between future out- 
comes for treatment group members and control group members understate the effects of re- 
ceiving the treatment, because some proportion of the treatment group did not, in actuality, re- 
ceive the treatment. For the present study, this implies that the observed effects of winning an 
SSC lottery understate the actual effects of enrolling in an SSC. 

In addition, about 25 percent of study participants who lose an SSC lottery (and thus 
become control group members) enroll in another SSC because HSAPS ultimately assigns 
them to one. For those students, losing a lottery for a particular SSC is not the same as not 
being assigned to any SSC. Hence, for control group members who are subsequently assigned 
to an SSC, the effect of winning an SSC lottery is not the same as the effect of being assigned 
to an SSC. 11 Indeed, it is not clear what the effect of winning an SSC lottery means for these 
students. 

Furthermore, about 10 percent of control group members enroll in an SSC through an 
avenue other than HSAPS. Both types of control group members who enroll in an SSC — those 
who arrive there through an HSAPS assignment and those who do not — experience SSCs, 
which further dilutes the difference in this experience between lottery winners and control group 



11 As noted earlier, the effect of being assigned to or offered a “treatment” is referred to as the effect of 
“intent to treat” in the statistics literature on causal inference. (See, for example, Angrist, Imbens, and Rubin, 
1996.) The effect of winning an SSC lottery is therefore not the same as the effect of intent to treat (or the 
effect of enrolling in an SSC). 
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members. This dilution causes the estimated effects of winning an SSC lottery to further unders- 
tate the effects of enrolling in an SSC. 

Because of the diluted contrast in SSC experiences between the two study groups and 
because the estimated effects of winning an SSC lottery cannot be interpreted in a meaningful 
way, it was necessary to convert them into estimates of something that does have a meaningful 
interpretation: the effects of enrolling in an SSC. Appendix A explains how this conversion was 
done using a well-known approach called “instrumental variables analysis.” The appendix also 
presents the assumptions upon which this analysis is based and demonstrates their plausibility 
for the present analysis. 

It is beyond the scope of this chapter to explain this analysis in detail. However, it is 
important to reiterate that it starts by estimating the effects of winning an SSC lottery based on 
existing data for all lottery winners and control group members (again, shown in Appendix B). 
Consequently, it does not drop any students with existing data from the analysis, and by not 
doing so it avoids the potential for “selection bias,” which can occur if students are selectively 
(as opposed to randomly) omitted from a treatment group or a control group. 12 Instead, the 
present analysis adjusts estimates of the effects of winning an SSC lottery for the proportion of 
lottery winners who do not enroll in an SSC and for the proportion of control group members 
who do enroll in an SSC. This adjustment corrects for the dilution that these sample members 
cause when looking at the difference between the two groups in their exposure to SSCs. 

Given the nature of the adjustment, the estimates of SSC enrollment effects that it pro- 
duces are for a subgroup of SSC lottery wimiers who are referred to throughout this report as 
“target SSC enrollees.” This label is used to denote that members of the subgroup are the target 
of estimation. The subgroup comprises about 58 percent of all SSC lottery wimiers, and all find- 
ings on SSC enrollment effects in the following chapters are for this subgroup. (See Appendix 
A for more details.) 

Size of the Sample, Its Data Availability, and Implications of 
Missing Data 

As noted, the present analysis is based on data for four annual cohorts of entering ninth- 
graders. Analyses of student progress in their first year of high school are based on data for all 
four cohorts; analyses of student progress in their second year are based on data for three co- 
horts; analyses of student progress in their third year are based on data for two cohorts; and ana- 
lyses of student progress in their fourth year are based on data for one cohort (the earliest one). 



12 Bloom (2006). 
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Table 2.2 reports the size of the randomized sample for each high school year and lists 
the percentage of sample members for whom follow-up data are available. Sample sizes are 
reported as numbers of student observations. The first and second colum ns in the table indicate 
the number of student observations for SSC lottery winners and control group members. This 
ranges from 12,978 for lottery winners and 17,981 for control group members in the first-year 
sample to 3,152 for SSC lottery winners and 5,131 for control group members in the fourth- 
year sample. 



New York City Small Schools of Choice 
Table 2.2 

Sample Sizes and Data Availability 



Number of Student Observations 


Percentage with Course Credits Follow-up Data 


Studv Sample 


SSC 

Lottery 

Winners 


Control 

Group 

Members 


SSC 

Lottery 

Winners 


Control 

Group 

Members 


Estimated 

Difference 


P-Value for 
Estimated 
Difference 


Year 1 of high school 
(4 cohorts) 


12,978 


17,981 


89.7 


89.8 


-0.1 


0.798 


Year 2 of high school 
(3 cohorts) 


10,377 


16,308 


83.3 


83.5 


-0.2 


0.755 


Year 3 of high school 
(2 cohorts) 


6,471 


11,513 


75.9 


76.5 


-0.5 


0.545 


Year 4 of high school 
(1 cohort) 


3,152 


5,131 


68.3 


69.4 


-1.1 


0.397 



SOURCES: MDRC's calculations use High School Application Processing System data from eighth-graders in 
2004-2005 to 2007-2008, as well as data from New York City Department of Education course credit and 
enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for the difference 
between SSC lottery winners and control group members are obtained from a regression of a given baseline 
characteristic on a series of indicator variables that identify each lottery plus an indicator variable that equals 1 for 
lottery winners and 0 for lottery losers. The coefficient on the latter indicator variable equals the difference in the 
mean baseline characteristic for lottery winners and control group members. The value for control group members 
equals the corresponding value for SSC lottery winners minus the estimated difference between lottery winners 
and control group members. To facilitate computation, all variables are centered on the mean value for the lottery 
they represent. This approach is equivalent to directly accounting for each lottery by adding a 0/1 indicator 
variable for it (Wooldridge, 2000). In some cases, rounding may cause slight discrepancies. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: ** = 

1 percent; * = 5 percent. 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 2005, 2006, 
2007, and 2008, respectively. 
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As described below, estimates of SSC effects are based on administrative data from the 
New York City Department of Education. These data are only available for students who enroll 
in a New York City public school. They are not available for students who move out of the city, 
transfer to a private or parochial school, or drop out of school. The third and fourth columns in 
the table indicate the percentage of SSC lottery wimiers and control group members for whom 
follow-up data are available. 13 

Note, in the last two columns of the table, that for all four years of high school, differ- 
ences in rates of data availability for SSC lottery winners and control group members are small 
and not statistically significant. Thus, rates of sample attrition are the same for SSC lottery win- 
ners and control group members. In addition, Appendix Tables C.l through C.4 demonstrate 
that for all four years of high school, mean background characteristics of SCC lottery winners 
with follow-up data are the same as those of control group members with follow-up data. This 
indicates that, although some members of the study sample drop out, the baseline characteristics 
of the two groups remain similar, which implies that attrition does not weaken the validity of 
estimates of SSC effects. 

Note next that the percentage of student observations with follow-up data declines from 
a high of almost 90 percent in the first year of high school to a low of just under 70 percent in 
the fourth year. Thus, the scope of the study sample narrows over time. However, the baseline 
equivalence between SSC lottery winners and control group members who have follow-up data 
remains strong (see Appendix C). This indicates that study findings are most likely valid for 
students with follow-up data. Missing data are only likely to affect the generalizability of these 
findings. 

Without follow-up data for all students, SSC effects cannot be estimated for the full 
study sample. Thus, the findings that are presented in this report are only for students who have 
follow-up data. However, studies such as this one generally consider how missing data for some 
students might affect the results for the full sample. Typically, this is done by assuming values 
for the missing data (“imputing” these values) and repeating the analysis with the assumed val- 
ues. While there are numerous approaches for imputation, they vary markedly in their com- 
plexity, transparency, and assumptions, and no consensus exists about which approach is most 
appropriate. Thus, in the absence of a compelling reason to use a particular approach, the analy- 
sis presented in this report uses two simple approaches to illustrate what SSC enrollment effects 
might possibly be for the full study sample. 



13 Simple percentages are reported for SSC lottery winners, and average percentages are reported for con- 
trol group members, weighted by the number of winners per lottery. This accounts for lottery differences in the 
ratio of winners to control group members. 
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The two imputation approaches, and the associated effects, are presented in Appendix 
D. The first — direct — approach makes the relatively conservative assumption that any student 
with missing data for an outcome was unsuccessful on that outcome. (For example, any student 
who is missing graduation data is assumed to have not graduated.) The second approach utilizes 
DOE discharge codes to distinguish between students who are identified as dropouts and all 
other students for whom data are missing. Students who are identified as dropouts are coded as 
not having graduated. Other students with missing data are randomly assigned a code of “grad- 
uated” or “not graduated” in proportion to the graduation rate for control group students who 
have follow-up data . 14 

As expected, estimated effects obtained using the preceding two imputation approaches 
compared with estimates of SSC effects on graduation rates without imputation (for students 
who were not missing follow-up data) are somewhat smaller and are almost identical for the 
two imputation approaches. However, all three estimates are positive and statistically signifi- 
cant, indicating that SSCs increase graduation rates. 

Remember, however, that findings based on imputation are speculative only. Hence, as 
noted above, this report focuses solely on estimates of SSC effects for students with follow-up 
data. 

The Data and Their Sources 

The primary sources of data for this report are infonnation from HSAPS and from pub- 
lic school records for individual students. This information was obtained from the New York 
City Department of Education. In addition, publically available data on school characteristics 
were obtained from New York State’s School Report Cards. Appendix E describes these data 
and their sources. 

HSAPS data were used to identify students who participated in an SSC lottery in or- 
der to determine the school to which they were assigned, to describe their background charac- 
teristics, and to compare background characteristics of SSC lottery winners and control group 
members. These data include student’s rank-ordered high school preferences and demograph- 
ic characteristics at the point of eighth-grade choice plus SSCs’ student priorities based on 
their geographic and “known” status. 

Students’ school records data were used to determine follow-up measures of academic 
progress, which are the basis for estimating SSC effects. This infonnation includes enrollment, 



14 Information on student discharge codes was not used as part of the primary estimation approach for the 
present analysis because this type of information is typically unreliable. 
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attendance, and course credits earned, plus state test scores and results of state Regents exami- 
nations. Eighth-grade standardized test scores in reading and math were obtained for baseline 
comparisons of SSC lottery winners and control group members. 

Publicly available data on school characteristics from New York State’s School Report 
Cards were used to compare SSCs with other public high schools in New York City. These data 
were also used to compare features of the schools attended by SSC lottery winners and control 
group members, such as their size, their number of years in existence, and characteristics of 
their teachers and students. 



Who Is Represented by the Findings? 

As noted, this analysis is based on data for students from 297 randomized lotteries. The 
lotteries were held for 105 of the 123 SSCs that were operating by fall 2008. These lotteries are 
spread across four annual cohorts of ninth-graders who entered high school between fall 2005 
and fall 2008. Estimates of effects of enrolling in an SSC are reported for the roughly 58 percent 
of SSC lottery wimiers who are referred to as “target SSC enrollees.” (Appendix A describes 
how this group is defined and identified.) Given the large size of this sample (about 7,559 stu- 
dents), findings for it are policy-relevant in their own right. 

Nevertheless, it is important to consider how well these findings generalize to all 42,528 
students who entered ninth grade at the 105 study SSCs during the period of analysis. This larg- 
er group of students, which is the most immediate and direct population of interest, is referred to 
as “all enrollees in the study SSCs.” To provide an even broader context, it is also useful to 
compare target SSC enrollees with all first-time ninth-graders in New York City public high 
schools. Although this study makes no pretense of generalizing findings to this much larger 
population, a comparison of the study sample with that larger population helps to place in con- 
text just who the study represents. 

Table 2.3 facilitates these comparisons. The first column reports mean background 
characteristics for target SSC enrollees. Appendix A describes how these characteristics are 
estimated. The second column reports this information for all enrollees in study SSCs, which 
is estimated from their individual data. The third column reports the same information for all 
first-time ninth-graders in New York City public high schools, which is also estimated from 
their individual data. 

Findings in the table indicate that target SSC enrollees are the same on average as all 
enrollees in the study’s SSCs in terms of every characteristic, with two exceptions. There is al- 
most no discemable difference in their race/ethnicity, gender, eligibility for free or reduced-price 
lunch (a proxy for low-income status), English language learner status, likelihood of being over- 
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New York City Small Schools of Choice 
Table 2.3 



Baseline Characteristics of Target SSC Enrollees, All HSAPS Enrollees 
in Study SSCs, and All First-Time Ninth-Grade Students in New York City: 
First Year of High School, Cohorts 1 to 4 



Characteristic (%) 


Target 

SSC 

Enrollees 


All HSAPS 
Enrollees in 
Study SSCs 


All First-Time 
Ninth-Grade 
Students in NYC 


Race/ethnicity 


Hispanic 


48.9 


48.4 


39.8 


Black 


43.7 


45.2 


34.2 


Other 


7.3 


6.4 


26.0 


Male 


47.9 


50.8 


51.3 


Eligible for free/reduced-price lunch 


83.2 


83.8 


74.9 


Special education 3 


6.7 


15.5 


14.0 


English language learner 


7.3 


8.1 


11.7 


Overage for 8th grade b 


21.2 


24.4 


21.7 


8th-grade reading proficiency 3 


Did not meet standards (level 1) 


7.0 


10.9 


10.2 


Partially met standards (level 2) 


62.9 


62.8 


51.7 


Fully met standards (level 3) 


29.3 


25.7 


34.8 


Met standards with distinction (level 4) 


0.8 


0.7 


3.3 


8th-grade math proficiency 3 


Did not meet standards (level 1) 


18.0 


22.4 


18.2 


Partially met standards (level 2) 


45.4 


44.8 


36.0 


Fully met standards (level 3) 


34.2 


30.9 


36.9 


Met standards with distinction (level 4) 


2.4 


1.9 


9.0 


Borough (home residence) 


Bronx 


54.9 


49.3 


22.9 


Brooklyn 


28.4 


30.9 


32.0 


Manhattan 


8.3 


11.4 


12.1 


Queens 


6.2 


7.4 


26.9 


Staten Island 


2.1 


0.9 


5.6 


Total number of students = 42,528 









SOURCES: MDRC's calculations use High School Application Processing System (HSAPS) and 
New York City Department of Education (DOE) state test data from eighth-graders in 2004-2005 to 
2007-2008, as well as data from DOE enrollment files from the 2005-2006 to 2008-2009 school 
years. 

NOTES: Appendix A describes how values in the column labeled "Target SSC Enrollees” are 
estimated. 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 
2005, 2006, 2007, and 2008, respectively. 

Previous year's enrollment files were used to determine whether or not a student was a first-time 
ninth-grader. 

(continued) 
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Table 2.3 (continued) 

a The target SSC enrollee sample includes special education students who can be taught in the 
regular classroom setting. Special education students classified by the DOE as requiring 
collaborative team teaching services or self-contained classes are not part of the sample but are 
enrolled in study SSCs and are thus included in the "All HSAPS Enrollees in Study SSCs" column. 

b Students are classified as "overage for eighth grade" if they were 14 or older on September 1 of 
the eighth-grade school year. 

c Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade 
level for state math and reading exams. Due to missing test scores, the sum of levels 1-4 may not 
add to 100 percent. 



age for grade, and (perhaps most important) their prior performance on standardized tests of 
reading and math administered by the State of New York during eighth grade. The first observed 
difference between the two groups is a six-point gap in the percentage of students who live in the 
Bronx (with the target SSC enrollees more likely to reside in the borough). The second differ- 
ence relates to the percentage of students with special education status. As a result of data col- 
lection issues, the HSAPS-based study sample includes only special education students who can 
be taught in a regular (mainstream) classroom setting. However, a broader group of special edu- 
cation students, including those requiring collaborative team teaching services or self-contained 
classes, are served by the study SSCs. This results in a noticeable difference in the proportion of 
special education students among target SSC enrollees (6.7 percent) versus among all enrollees 
in the study SSCs (15.5 percent). 

Thus, although it is never possible to know for sure whether a study’s findings general- 
ize adequately to a given population (because a study’s sample can differ in unobserved ways 
from the population), it is likely that findings for target SSC enrollees represent the high school 
experiences of most students who enrolled in 105 of the 123 SSCs that existed (or came into 
being) during the present analysis period. 

Now compare the sample of target SSC enrollees with the population of all first-time 
ninth-graders in New York City public high schools, as shown in Table 2.3. Note first that a 
much greater percentage of sample members are black or Hispanic (92.6 percent versus 74 per- 
cent). Thus, as intended, SSCs are attracting large numbers of students of color. Note next that a 
somewhat greater percentage of target SSC enrollees are eligible for free or reduced-price 
lunches than is the case for all entering ninth-graders in New York City (83.2 percent versus 
74.9 percent). Thus, also as intended, SSCs are attracting large numbers of low-income stu- 
dents. Note next that differences between the percentage of target SSC enrollees who are Eng- 
lish language learners, designated for special education, and/or overage for grade are relatively 
small (especially for the latter group). Thus, SSCs are not excluding these important subgroups 
of children. Note next that the prior proficiency in reading and math for the study sample is, on 
balance, relatively similar to that of incoming ninth-graders citywide. If anything, students in 
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the study sample are somewhat less proficient, but this difference is neither large nor consistent 
across the four proficiency levels reported by New Y ork State. However, both in the sample and 
citywide, well over half of incoming ninth-graders were performing below grade level in read- 
ing and in math as they entered high school . 15 Thus, as intended, SSCs are attracting large num- 
bers of students who are struggling in school. 

The only other marked difference between target SSC enrollees and all incoming ninth- 
graders citywide is where they live. For example, target SSC enrollees are much more likely to 
be from the Bronx and much less likely to be from Queens, which is not surprising given the 
locations of the SSCs. 

Overall, then, it appears that as they enter high school, target SSC enrollees “look” very 
much like all enrollees at SSCs in the study and represent in large numbers (as intended) stu- 
dents of color, students living in poverty, and students who are struggling in school. It is the 
high school experiences of these students that the findings in the next chapter represents. 



15 Over the course of the study period, citywide averages of prior proficiency of incoming ninth-graders in- 
dicate that 61.9 percent were performing below grade level in reading and 54.2 percent were performing below 
grade level in math. For the study sample, 69.9 percent were performing below grade level in reading and 63.4 
percent were performing below grade level in math. 
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Chapter 3 



The Effects of Enrolling in 
Small Schools of Choice on 
Students’ High School Academic Progress 



This chapter reports the estimated effects of enrolling in a small school of choice (SSC) 
for target SSC enrollees. 1 The chapter briefly compares the key features of the SSCs that were 
attended by the target SSC enrollees with those of schools attended by their control group coun- 
terparts, noting in particular that target SSC enrollees attend schools that are much newer — 
they were founded in 2002 or later — and smaller than those attended by their control group 
counterparts. 

The resulting findings indicate that the SSCs have a strong and sustained positive im- 
pact on student achievement, leading to improved graduation rates: 

• Target SSC enrollees have a more successful academic transition into high 
school during ninth grade than do their control group counterparts. 

• The effect of a more successful transition into high school is observed for a 
broad range of students who differ in terms of prior academic proficiency, 
race/ethnicity, gender, eligibility for free or reduced-price lunch, and stated 
preference for their SSC. 

• These SSC effects are sustained during students’ second and third years of 
high school and culminate in higher rates of graduation by their fourth year. 



Schools Attended by Target SSC Enrollees and by Their 
Control Group Counterparts 

To properly interpret estimates of SSC effects, it is necessary to understand the differ- 
ences that exist between schools attended by target SSC enrollees and those attended by their 
control group counterparts. It is these differences that produce the effects observed. 

As explained earlier, New York City’s High School Application Processing System 
(HSAPS) and the lotteries it created yielded an unprecedented opportunity to launch a rigorous 

As explained in Chapter 2, target SSC enrollees make up the subgroup of SSC lottery winners who are 
represented by adjusting the estimated effects of winning an SSC lottery (presented in Appendix Tables B. 1 
through B.4) for the proportion of winners who do not enroll in an SSC and the proportion of their control 
group counterparts who do. 
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and credible study of a large number of schools at the system level. At the same time, the retro- 
spective nature of the study means that it was not possible to obtain information on many poten- 
tially important features of schools, such as instructional practices, school climate, and qualitative 
measures of student experience. In addition, the study’s control group members attend a broad 
range of high schools located throughout New York City, not just schools that are large or old or 
underperforming, making it difficult to describe those schools succinctly. Still, the scale of the 
study in terms of both number of students and the proportion of schools (105 SSCs out of 123 
existing SSCs), the level of diversity it covered, the existence of randomization, the study popu- 
lation of historically underserved students, and the rapidity with which the reforms were imple- 
mented all contributed to the relevance and reliability of the findings presented here. 

Table 3.1 compares features of the schools attended by the average target SSC enrollee 
with those attended by the average control group counterpart. The table focuses first on the de- 
fining features of SSCs: (1) their age, or when they were established, keeping in mind that New 
York City began creating SSCs in 2002; and (2) their size, which is noteworthy because SSCs 
are intended to offer students a small, personalized environment. 

As expected, high schools attended by control group counterparts are much older than 
those attended by target SSC enrollees. Only 9.0 percent of control group members attended a 
high school that had opened since 2002. An additional 15.8 percent attended a high school that 
was restructured in 2002 or thereafter into small learning communities (SLCs) and/or career or 
technical education programs. The remaining 66.8 percent of control group counterparts at- 
tended high schools that were opened before (often well before) 2002 and had not been re- 
structured recently. 

Also as expected, high schools attended by target SSC enrollees are much smaller than 
those attended by control group counterparts. For example, there were 129 students in the aver- 
age target SSC enrollee’s ninth-grade class versus 635 for the average control group counter- 
part. Because some of the city’s large, comprehensive high schools have been restructured into 
small learning communities, potentially providing a “smaller” experience than a school’s size 
alone would indicate, this table separately indicates the 12.5 percent of control group counter- 
parts enrolled in such a school. 2 



‘It could be argued that in high schools with small learning communities, a new student does not expe- 
rience the hill school population of ninth-graders but rather the student population of the small learning com- 
munity. However, because reliable data on the number of students enrolled in individual SLCs does not exist, 
those schools’ trill ninth-grade population is used to calculate the average ninth-grade size for control group 
counterparts. In an effort to address the potentially more personalized experience provided by schools with 
SLCs, when described in terms of their total size, SLCs are not characterized as simply small, medium, or 
large, but instead as medium or large with SLCs. 
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New York City Small Schools of Choice 
Table 3.1 



Characteristics of Schools Attended by Target SSC Enrollees Compared 
with Characteristics of Schools Attended by Their Control Group Counterparts: 
First Year of High School, Cohorts 1 to 4 





Schools attended bv 








Target 


Control 




P -Value for 




SSC 


Group 


Estimated 


Estimated 


School Characteristic 


Enrollees 


Counterparts 


Difference 


Difference 


School ase (%) 










School opened since 2002 


100.0 


9.0 


91.0 ** 


0.000 


School was reformed/restructured since 2002 


0.0 


15.8 


-15.8 ** 


0.000 


School established before 2002 


0.0 


66.8 


-66.8 ** 


0.000 


School size 11 










Number of students enrolled in ninth grade 


129.0 


634.9 


-505.9 ** 


0.000 


Small - 550 students or less (%) 


100.0 


20.7 


79.3 ** 


0.000 


Medium - 55 1-1,400 students (%) 


0.0 


20.3 


-20.3 ** 


0.000 


Large - more than 1 ,400 students (%) 


0.0 


32.4 


-32.4 ** 


0.000 


Medium and large with small learning communities (%) 


0.0 


12.5 


-12.5 ** 


0.000 


School's 9th grade population (%) 










Race/ethnicity 










Hispanic 


48.9 


51.8 


-3.0 ** 


0.000 


Black 


43.1 


40.2 


2.9 ** 


0.000 


Other 


8.1 


8.0 


0.1 


0.704 


Eligible for free/reduced-price lunch 


83.0 


84.7 


-1.7 ** 


0.000 


Special education 


14.9 


16.7 


-1.8 ** 


0.000 


English language learners 


7.9 


12.9 


-5.0 ** 


0.000 


Scored at or above 8th-grade level in reading 11 


28.6 


23.3 


5.3 ** 


0.000 


Scored at or above 8th-grade level in math b 


35.4 


30.5 


5.0 ** 


0.000 


Overage for 8th grade c 


21.0 


25.9 


_4 9 ** 


0.000 


9th-grade repeaters 


13.2 


28.3 


-15.1 ** 


0.000 


Teacher characteristics (%) 










Less than 3 years of teaching experience 


37.8 


22.8 


15.0 ** 


0.000 


Doctorate or master's degree plus 30 hours 


21.7 


30.9 


-9.2 ** 


0.000 


Total number of student observations = 29,81 1 



(continued) 
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Table 3.1 (continued) 

SOURCES: MDRC calculations use High School Application Processing System and New York City 
Department of Education (DOE) state test data from eighth-graders in 2004-2005 to 2007-2008, data from 
the New York State Report Card for 2002-2003 to 2007-2008 school years, DOE enrollment and course 
credit files for the 2005-2006 to 2008-2009 school years, and DOE school-level administrative records for 
the 2002-2003 to 2008-2009 school years. 

NOTES: This table presents the estimated differences for students who have follow-up course credits data. 
Appendix A describes how values in the column labeled "Target SSC Enrollees" are estimated. Appendix A 
also describes how values in the column labeled "Estimated Difference" are estimated. Values in the column 
labeled "Control Group Counterparts" are differences between corresponding values in the first and third 
columns. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: 
** = 1 percent; * = 5 percent. 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 2005, 
2006, 2007, and 2008, respectively. 

a Roughly 1 4 percent of the control group counterparts enrolled in schools that are not easily defined by 
their size: 12 percent enrolled in middle/high schools and 2 percent enrolled in specialized high schools. 

b Students scoring at proficiency levels 3 and 4 are considered to be performing at or above grade level for 
state math and reading exams. 

c Lottery participants are classified as "overage for eighth grade" if they were 14 or older on September 1 
of the eighth-grade school year. 



Corresponding differences exist in the total sizes of schools attended by the two groups 
(for grades 9 through 12). Every target SSC enrollee attended a school with 550 or fewer stu- 
dents, whereas control group counterparts attended schools that range from 550 or fewer stu- 
dents (for 20.7 percent) to between 551 and 1,400 students (for 20.3 percent) to over 1,400 stu- 
dents (for 32.4 percent). 

The next part of Table 3.1 describes the ninth-grade student populations of the schools 
that the two groups attend — that is, not just the members of the study sample, but the full 
ninth-grade student population at each school (that is, their peers). This information makes it 
possible to compare student peers of target SSC enrollees with those of control group counter- 
parts. It is important to note that these findings represent features of the schools attended by 
sample members, not characteristics of the sample members themselves. 

In terms of socioeconomic characteristics, ninth-grade peers of the two groups are quite 
similar. For example, 48.9 percent of peers for the average target SSC enrollee are Hispanic and 
43.1 percent are black versus 51.8 and 40.2 percent, respectively, for the average control group 
counterpart. In addition, 83.0 percent of the peers of target SSC enrollees are eligible for free or 
reduced-prices lunches, 14.9 percent are designated for special education, and 7.9 percent are 
English language learners, versus 84.7, 16.7, and 12.9 percent, respectively, for control group 
counterparts. 
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However, the ninth-grade peers of target SSC enrollees appear somewhat stronger 
academically than the peers of control group counterparts. This difference requires some 
explanation, because its magnitude (and thus interpretation) depends on whether it is 
viewed through the lens of prior characteristics of current ninth-grade students (which de- 
scribe them before entering high school) or current characteristics (which describe them 
after entering high school). In terms of pre-existing characteristics, the ninth-grade peers of 
target SSC enrollees have a 5 percentage point academic advantage over the ninth-grade 
peers of control group counterparts. They are 5.0 and 5.3 percentage points more likely to 
have scored at or above the eighth-grade level on their New York State tests of reading and 
math, respectively; they are 4.9 percentage points less likely to have been overage for 
eighth-grade (that is, before entering high school); and they are 5.0 percentage points less 
likely to be English language learners. However, in terms of current characteristics, the 
ninth-grade peers of target SSC enrollees are 15.1 percentage points less likely to have re- 
peated ninth grade than are the ninth-grade peers of control group counterparts. 

The simplest, most direct, and thus most plausible explanation for this marked disparity 
in differences is that SSCs facilitate high school academic progress more effectively than do the 
schools attended by control group members. This conclusion is confirmed by the fact demon- 
strated below that SSCs increase the percentage of ninth-graders who make adequate progress 
toward graduation by 10.0 percentage points (which implies that SSCs reduce the likelihood of 
repeating ninth grade by about the same margin). Consequently, most of whatever advantage 
target SSC enrollees experience because of their ninth-grade peers is probably caused by what 
SSCs do for students, not by the type of students they attract. 

The last two lines of Table 3.1 provide a limited comparison of the teachers of target 
SSC enrollees with those of their control group counterparts. As can be seen, teachers of target 
SSC enrollees have fewer years of experience and are less likely to have graduate school cre- 
dits or degrees than are teachers of control group counterparts. 3 This finding aligns with the 
prevailing wisdom that SSCs generally attracted teachers who were new to the system or the 
profession. However, it is not clear what these differences mean in terms of students’ class- 
room experiences. 



Effects of SSC Enrollment on High School Academic Success 

It is well known that making a successful transition into high school is a critical step 
toward graduation. For example, the Consortium on Chicago School Research found that high 



’These comparisons are made in terms of school-level teacher characteristics because it was not possible 
to obtain information on specific teachers for each sample member. 
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school students who are on track toward graduation by the end of their first year (based on an 
“on-track indicator” that is replicated for the purposes of the present study) are three and a half 
times more likely to graduate in four years than are other students. 4 Conversely, a substantial 
body of literature documents that for many low-performing youths, the transition into high 
school is marked by disengagement and declining motivation, which are precursors to school 
dropout. 5 For these reasons, the present chapter begins its story with students’ academic 
progress in the first year of high school. 

Of course, the bottom-line indicator of a high school’s success is its students’ ability 
to graduate and advance to employment that is economically and personally fulfilling. This 
path might include further education after high school (which is becoming increasingly im- 
portant as workplace skills become more sophisticated) or it might involve an immediate 
move into the job market. But without a high school diploma, these paths are not readily 
available. This fact of modern life makes it all the more important that a solution be found to 
the chronically high dropout rates that exist in many urban high schools, especially for stu- 
dents who live in poverty and are often black or Hispanic. As noted, New York City created 
SSCs with the explicit goal of addressing this very pressing problem. Following a discussion 
of the transition into high school, this chapter presents the effects of enrolling in an SSC on 
progress toward graduation and the attainment of a recognized diploma. The results indicate 
that SSCs are in fact making progress toward their stated goal. 

Average Effects During the First Year of High School 

Table 3.2 presents estimates of SSC effects on students’ likelihood of being on track 
toward graduation at the end of their first year of high school plus estimates of SSC effects on 
students’ attendance during that year. (Box 3.1 describes those outcomes.) Findings in the first 
row of the table indicate that by the end of the year, 58.5 percent of target SSC enrollees were 
on track to graduate in four years compared with 48.5 percent of their control group counter- 
parts. The 10.0 percentage point difference is both large in size (and thus policy-relevant) and 
statistically significant (and thus unlikely to represent a chance result due to random error). 6 

This first-year SSC effect helps to explain the apparent transformation in peer academic 
differences discussed alongside Table 3.1 above because (1) it determines the likelihood of re- 
peating ninth grade, and (2) its size is equivalent to the magnitude of the transformation. Hence, 



4 Allensworth and Easton (2005). 

5 National Research Council (2004). 

6 As noted in Chapter 2, this result for the full study sample is virtually identical to its counterpart for the 
three-fourths of sample members who were not in an HSAPS lottery for a preferred school. The estimated ef- 
fect for the subsample is an increase of 9.78 percentage points in the likelihood of making adequate progress 
toward graduation in ninth grade (p-value = <0.0001). 
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New York City Small Schools of Choice 
Table 3.2 

Estimated Effects of SSC Enrollment on the Transition into High School: 
First Year of High School, Cohorts 1 to 4 





Target 


Control 




Effect Size 


P-Value for 




SSC 


Group Estimated 


(Standard 


Estimated 


Outcome 


Enrollees Counterparts 


Effect 


Deviation) 


Effect 


Course credits 












9th-grade on-track indicator 3 (%) 


58.5 


48.5 


10.0 


** 


0.000 


Earned 10 or more credits 


73.1 


62.3 


10.8 


** 


0.000 


Failed more than 1 semester of a core subject 


39.0 


46.8 


-7.8 


** 


0.000 


Total credits earned toward graduation 13 


11.3 


10.4 


0.9 


** 0.21 


** 0.000 


Attendance (%) 












Overall attendance rate 


87.3 


86.5 


0.8 




0.094 


Regular attendance rate (90 percent or higher) 


60.5 


55.1 


5.4 


** 


0.000 


Total number of student observations = 29,81 1 



SOURCES: MDRC's calculations use High School Application Processing System data from eighth-graders 
in 2004-2005 to 2007-2008, as well as data from New York City Department of Education attendance, course 
credits, and enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: This table presents the estimated effects for students who have follow-up course credits data. 
Appendix A describes how values in the column labeled "Target SSC Enrollees" are estimated. Appendix A 
also describes how values in the column labeled "Estimated Effect" are estimated. Values in the column 
labeled "Control Group Counterparts" are differences between corresponding values in the first and third 
columns. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated as: ** = 
1 percent; * = 5 percent. 

The estimated effect size is calculated as a proportion of the standard deviation of the outcome for control 
group counterparts. 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 2005, 

2006, 2007, and 2008, respectively. 

a The on-track composite measure indicates whether students earned at least 10 credits in their first year of 
high school and had no more than one semester of failure in a core subject in that school year (English, math, 
science, and social studies). 

b The "total credits earned toward graduation" measure is the aggregate number of course credits earned 
toward fulfilling the New York State graduation requirements. The credit requirements are as follows: 31 
core subject credits, including 8 credits each of English and social studies; 6 credits each of math and 
science; 2 credits of arts; 1 credit of health; and 13 additional credits, including 4 credits of physical 
education, 2 credits of a foreign language, and 7 credits of electives. 



41 







New York City Small Schools of Choice 
Box 3.1 

Definitions of Outcomes: Transition Into and Progress 
Through High School 

Course Credits 

• Ninth-grade on-track indicator: The on-track indicator, a composite measure, indi- 
cates whether students earned at least 10 credits in their first year of high school and had 
failed no more than one semester of a core subject (English, math, science, and social 
studies) in that school year. 

• Failed more than one semester of a core subject: This measure indicates that a stu- 
dent has failed more than one semester in a core subject. 

• Earned 10 or more credits: This measure indicates that a student has earned at least 10 
credits by the end of the first year of high school. It acts as a proxy for fulfilling the re- 
quirements for promotion to tenth grade in the following year. 

• Earned 20 or more credits: This measure indicates that a student has earned at least 20 
credits by the end of the second year of high school. It acts as a proxy for fulfilling the 
requirements for promotion to eleventh grade in the following year. 

• Earned 30 or more credits: This measure indicates that a student has earned at least 30 
credits by the end of the third year of high school. It acts as a proxy for fulfilling the re- 
quirements for promotion to twelfth grade in the following year. 

• Total credits earned toward graduation: In order to graduate, New York State re- 
quires that students earn 44 credits: 31 core subject credits, including 8 credits each of 
English and social studies; 6 credits each of math and science; 2 credits of arts; 1 credit 
of health; and 13 additional credits, including 4 credits of physical education, 2 credits 
of a foreign language, and 7 credits of electives. 

Attendance 

• Overall attendance rate: The total number of days a student was present during the 
school year, divided by the total number of days a student was enrolled. 

• Regular attendance rate (90 percent or higher): A binary measure for a student’s at- 
tendance rate being greater than 90 percent, or attended at least 9 out of every 10 days 
enrolled, over the course of the school year. 

Regents Exams 

• Total Regents exams passed toward graduation: In order to receive a New York State 
diploma, students must pass the following core subject Regents exams: English Lan- 
guage Arts, Math A, U.S. History, Global History, and one of the science exams (Chemi- 
stry, Physics, Earth Science, Biology, or Living Environment). The “total Regents exams 
passed toward graduation” measure counts the number of required Regents exams that a 
student has passed with a score of 65 or above. 
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most of whatever advantage target SSC enrollees experience because of academically stronger 
peers is probably caused by what SSCs do for these peers, not by what these peers bring to SSCs. 

The “on-track indicator” in Table 3.2 consists of the two components listed below it. 
The first component is whether or not a student earns at least 1 0 credits during the first year of 
high school. Enrolling in an SSC increases the likelihood of doing so by 10.8 percentage 
points — from 62.3 percent to 73.1 percent. To place this finding in context, note that, in 
terms of student performance, it is the equivalent of an improvement in a school’s position 
from the twenty-eighth percentile to the fifty-third percentile among all public high schools in 
New York City. 7 In other words, it is equivalent to almost a full quartile difference in the dis- 
tribution of the city’s public high schools, spanning roughly 100 schools. 8 

The second component of the on-track indicator is whether or not students fail more 
than a single core course (Box 3.1 lists core subjects) during one semester. Findings in the table 
indicate that enrolling in an SSC reduces the likelihood of such failure by 7.8 percentage points 
— from 46.8 percent to 39.0 percent. 

The net result of the preceding two SSC effects is that during the first year of high 
school, target SSC enrollees earn almost one full credit more (0.9 credit) toward graduation than 
do their control group counterparts. 

Two measures are used to examine SSC effects on student attendance (second panel of 
Table 3.2), which is a rough proxy for student engagement. Findings for the first measure indi- 
cate that enrolling in an SSC has at most a small effect on overall average attendance for ninth 
grade. However, findings for the second measure indicate that enrolling in an SSC increases the 
percentage of ninth-graders who attend school regularly — that is, they attend for at least 9 out 
of 10 days enrolled — by 5.4 percentage points. This evidence suggests that SSCs promote 
greater student engagement. 

Effects for Student Subgroups During the First Year of High School 

Table 3.3 demonstrates that students from many different subgroups experience positive 
SSC enrollment effects during the first year of high school. For this analysis, an estimate of the 
effect of SSC enrollment on the “on track to graduate” measure described above is reported for 
each subgroup. Results are positive for all subgroups and statistically significant for all but the 



7 School performance percentiles are based on publicly available 2007-2008 DOE Report Card data. 

8 Information on the distribution of school performance for this measure was obtained from 2007-2008 
Report Card data produced by the New York City Department of Education (DOE). Such publically available 
information is not available for the on-track indicator. 
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New York City Small Schools of Choice 
Table 3.3 



Variation in Effects of SSC Enrollment on Ninth-Grade On-Track Indicator, 
by Student Characteristics: First Year of High School, Cohorts 1 to 4 



Student Characteristic 


Estimated 

Effect 


P- Value for 
Estimated 
Effect 


8th-srade readme Droflciencv 11 






Did not meet standards (level 1) 


10.3 * 


0.026 


Partially met standards (level 2) 


12.0 ** 


< 0.001 


Fully met standards (level 3) 


11.9 ** 


< 0.001 


Met standards with distinction (level 4) 


- 


- 


8th-srade math Droflciencv 11 






Did not meet standards (level 1) 


10.0 ** 


0.001 


Partially met standards (level 2) 


10.8 ** 


<0.001 


Fully met standards (level 3) 


15.2 ** 


<0.001 


Met standards with distinction (level 4) 


- 


- 


Low-income status 






Eligible for free/reduced-price lunch 


12.3 ** 


< 0.001 


Not eligible for free/reduced-price lunch 


8.2 ** 


< 0.001 


Race/ethnicitv. bv sender 






Black male 


8.5 ** 


0.001 


Hispanic male 


7 7 ** 


0.001 


Other male 


7.4 


0.118 


Black female 


13.5 ** 


< 0.001 


Hispanic female 


12.0 ** 


< 0.001 


Other female 


9.7 


0.062 


Choice level fof 12) at which enrollee DarticiDated in lotterv 






1st choice 


7.1 * 


0.011 


2nd-3rd choices 


11.5 ** 


< 0.001 


All other choices 


11.6 ** 


< 0.001 



(continued) 
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Table 3.3 (continued) 



Student Characteristic 


Estimated 

Effect 


P-Value for 
Estimated 
Effect 


Known / not known HSAPS status 1 " 






Known to SSC 


74 ** 


0.001 


Not known to SSC 


11.8 ** 


< 0.001 


Cohort 






Cohort 1 (2004-2005) 


11.9 ** 


< 0.001 


Cohort 2 (2005-2006) 


8.2 ** 


< 0.001 


Cohort 3 (2006-2007) 


11.1 ** 


< 0.001 


Cohort 4 (2007-2008) 


10.9 ** 


< 0.001 



SOURCES: MDRC's calculations use High School Application Processing System (HSAPS) and New York Cifi 
Department of Education (DOE) state test data from eighth-graders in 2004-2005 to 2007-2008, as well as data 
from DOE course credits and enrollment files from the 2005-2006 to 2008-2009 school years. 

NOTES: This table presents the estimated effects for students who have follow-up course credits data. Each 
panel in this table divides students into subgroups based on a given characteristic. Within each subgroup a two- 
tailed t-test was applied to the estimated effect, with statistical significance levels indicated as: ** = 1 percent; * 
= 5 percent. An F-test was used to assess the statistical significance between subgroups; none of the subgroup 
differences in this table is statistically significant. 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 2005, 2006, 
2007, and 2008, respectively. 

Effects on level 4 math and reading proficiency subgroups are not estimable due to small sample sizes. 

a Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade level for state 
math and reading exams. 

b "Known / not known HSAPS status" is a measure of whether the SSC to which a student was assigned 
indicated that the student was known to the school. Known status was collected consistently in the HSAPS data 
for cohorts 2, 3, and 4, but not for cohort 1, which is thus not included in these calculations. 



two smallest. In addition, few subgroup differences are statistically significant. What this means 
is that SSCs are having a strong positive effect for all the subgroups, and that the strength of 
those effects is of similar magnitude. 

Consider first the findings for subgroups defined by students’ prior academic proficien- 
cy, as measured by their eighth-grade state test scores in reading and math. Recall from Chapter 
2 that New York State reports these results in four levels. Levels 1 and 2 (did not meet or par- 
tially met standards) represent student performance that is below grade level, and levels 3 and 4 
(fully met standards or met standards with distinction), the two highest levels, represent student 
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performance that is at or above grade level. Since very few sample members score in the top 
level, no estimates of SSC effects are reported for them. 

Results for all of these prior proficiency subgroups indicate that SSCs improve the tran- 
sition into high school. Estimates range from a 10.0 point to a 15.2 point increase in the percen- 
tage of students who are on track toward graduation. Thus, SSCs improve the transition into 
high school for students at widely varying levels of prior academic proficiency. 

Consider next the results for subgroups of students defined by an indicator of low- 
income status: their eligibility for free or reduced-price lunches. Once again, there are consis- 
tently positive effects for each subgroup. The roughly four out of five target SSC enrollees who 
are eligible for this government assistance experience a 12.3 percentage point gain in the like- 
lihood of being on track toward graduation. The one out of five target SSC enrollees who are 
not eligible experience an 8.2 percentage point gain. The difference between these two esti- 
mates is not statistically significant — meaning that regardless of the income level of the target 
SSC enrollee, the “on track for graduation” measure improved. 

Findings for subgroups defined in terms of students’ race/ethnicity are also consistently 
positive and do not show statistically significant variation overall. However, the pattern of these 
results is somewhat complex. On the one hand, they corroborate much past research that finds 
that educational initiatives benefit black and Hispanic females by more than they do black and 
Hispanic males. SSC enrollment effects are 13.5 and 12.0 percentage points for black and His- 
panic females, respectively, versus 8.5 and 7.7 percentage points for black and Hispanic males. 
In addition, the overall difference in effects between females and males among these students of 
color is statistically significant. 9 On the other hand, the results stand in stark contrast to much 
past research that indicates that it is very difficult to improve educational outcomes for black 
and Hispanic males. In other words, black females did better than black males, but unlike past 
study results, the black males did improve. 

The next results in the table are for student subgroups defined in terms of the way each 
student ranked the SSC to which HSAPS assigned him. 10 Hie estimated effect on progress to- 
ward graduation is 7.1 percentage points for students who were assigned to a first-choice SSC, 

11.5 percentage points for students who were assigned to a second- or third-choice SSC, and 

1 1.6 percentage points for students who were assigned to an SSC that was their fourth through 
twelfth choice. The overall variation in these estimates is not statistically significant, although 



9 The difference between estimates for black and Hispanic females and black and Hispanic males is statis- 
tically significant (p-value = 0.037), although given the many significance tests for subgroup findings in Table 
3.3, that difference may have occurred by chance as a result of random en'or. 

’’’Although by necessity these subgroups are defined in terms of students’ rank-ordered choice for the SSC 
to which they were assigned, all results are reported in terms of effects of enrolling in an SSC. 
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the difference between estimates for first-choice students and all others is significant. 11 Never- 
theless, SSCs had a positive effect on students’ academic transition into high school regardless 
of the choice level at which students had ranked the SSC to which they were assigned. 

The subgroups in the next panel of Table 3.3 distinguish between students who were 
known and those who were not known to the SSC to which HSAPS assigned them. 12 As noted, 
a student can become known to an SSC in many ways, including contacting it in person or by 
telephone, visiting it, and/or meeting with its representative at a high school fair. While there are 
many hypotheses about why and how students become known to a school, it is likely that for 
some students and families this is the result of taking the initiative and expending the effort to 
contact and learn about the school. Findings in the table indicate that students who were known 
to their SSC had an SSC enrollment effect of 7.4 percentage points and students who were not 
known had an estimated effect of 1 1.8 percentage points. 13 Hie difference between these two 
estimates is not statistically significant; in other words, both students who were known to their 
SSC and students who were not known experienced a positive SSC enrollment effect. 

The last set of student subgroups in the table comprises the four annual cohorts in the 
study sample. As noted, the first of these cohorts entered ninth grade in fall 2005 and the last 
entered in fall 2008. Estimates of SSC effects range from 8.2 to 11.9 percentage points and do 
not show any statistically significant variation, even though many things were changing rapidly 
in the New York City public school system as these cohorts were entering high school. 

On balance, then, estimated ninth-grade effects of enrolling in an SSC are robust across 
many student subgroups. This result indicates that SSCs are not just effective for certain types 
of students but rather are effective for a broad range of students who are interested enough to 
list an SSC among their 12 high school choices. 

Average Effects During the Second and Third Years of High School 

This section follows students’ academic progress through their second and third years 
of high school. To maximize precision and generalizability, findings are presented first for the 
largest samples possible. Table 3.4 presents the findings for Year 2 of high school based on data 
for three cohorts. Table 3.5 presents the findings for Year 3 of high school based on data for two 



1 1 Given the many significance tests for subgroup findings in Table 3.3, it is possible that the statistical sig- 
nificance of the difference between estimates for students enrolled in their first-choice schools and all others ip- 
value = 0.034) occurred by chance due to random error. 

12 These subgroups are also defined in terms of students’ rank-ordered choice for the SSC to which they 
were assigned, even though results are reported in terms of effects of enrolling in an SSC. 

1 'Known status was collected consistently in the HSAPS data for cohorts 2, 3, and 4, but not for cohort 1, 
which is thus not included in these calculations. 
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cohorts. Because these samples differ from each other and from the four-cohort sample used to 
estimate first-year SSC effects, it is not possible to use their findings to study the trajectory over 
time of SSC effects for a given group of students. Table 3.6 makes this possible by presenting 
selected findings by high school year for the one cohort for which this information is available. 
This more flexible analysis is limited, however, in terms of its precision and generalizability. 

Consider the second-year findings in Table 3.4. By that time, 69.4 percent of target SSC 
enrollees (from three cohorts) had earned 20 or more credits toward graduation as opposed to 
58.3 percent of control group counterparts. The 11.1 percentage point difference between the 
two groups reflects the magnitude of the difference in their likelihoods of promotion to eleventh 
grade. 14 In addition, by the end of their second year of high school, target SSC enrollees had 
accumulated an average of 22.3 credits toward graduation as opposed to 19.8 for control group 
counterparts, for a difference of 2.6 credits. 

The next panel in Table 3.4 reports an estimate of the SSC enrollment effect on the aver- 
age number of New York State Regents examinations passed by the end of students’ second year 
of high school. Passing a specified set of these examinations is required for students to graduate 
from high school with a New York State diploma. As can be seen, SSCs have a positive effect on 
this important outcome. However, interpreting SSC effects on it at an early point in students’ 
high school careers is difficult because schools’ philosophies and policies vary with respect to the 
appropriate time to administer Regents exams. Some schools administer these exams as early as 
possible, whereas other schools delay their administration as a conscious educational strategy. 
Therefore, it is not when students take and pass these examinations that counts, but, rather, 
whether they do so by their fourth year of high school, in order to graduate on time. 

The findings in the bottom panel of Table 3.4 focus on students’ attendance during their 
second year of high school. They indicate that SSCs increase the average overall attendance rate 
by 2.4 percentage points (from 82.5 percent for control group counterparts to 84.8 percent for 
target SSC enrollees) and increase the percentage of students who attend school regularly — 
that is, at least 90 percent of the time — by 6.2 percentage points (from 49.0 percent for control 
group counterparts to 55.2 percent for target SSC enrollees). Hence, SSCs continue to increase 
students’ engagement during their second year of high school. 

Now consider the third-year findings for two cohorts in Table 3.5. 15 These findings in- 
dicate that (1) SSCs increase the percentage of students earning 30 or more credits by 7.1 per- 
centage points; (2) SSCs increase the average number of credits earned toward graduation by 
2.4 credits; and (3) SSCs increase the average number of Regents examinations passed by 0.11 

14 Eaming 10 or more credits each year is generally considered as being on track for grade-level promotion. 
l5 Third-year findings in Table 3.5 should not be compared with second-year findings in Table 3.4 or with 
first-year findings in Table 3.2 because they represent different samples. 
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New York City Small Schools of Choice 
Table 3.4 



Estimated Effects of SSC Enrollment on Progress Toward Graduation: 
Second Year of High School, Cohorts 1 to 3 





Target 


Control 




Effect Size 


P-Value for 




SSC 


Group Estimated 


(Standard 


Estimated 


Outcome Enrollees 


Counterparts 


Effect 


Deviation) 


Effect 


Course credits 












Earned 20 or more credits (%) 


69.4 


58.3 


11.1 ** 




0.000 


Total credits earned toward graduation 3 


22.3 


19.8 


2.6 ** 


0.31 


** 0.000 


Regents exams 












Total Regents exams passed toward graduation 13 


1.5 


1.4 


0.1 * 


0.06 


* 0.032 


Attendance (%) 












Overall attendance rate 


84.8 


82.5 


2.4 ** 




0.000 


Regular attendance rate (90 percent or higher) 


55.2 


49.0 


6.2 ** 




0.000 


Total number of student observations = 21,822 



SOURCES: MDRC's calculations use High School Application Processing System data from eighth-graders 
in 2004-2005 to 2006-2007, as well as data from New York City Department of Education attendance, 
course credits, Regents exam, and enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: This table presents the estimated effects for students who have follow-up course credits data. 
Appendix A describes how values in the column labeled "Target SSC Enrollees" are estimated. Appendix A 
also describes how values in the column labeled "Estimated Effect" are estimated. Values in the column 
labeled "Control Group Counterparts" are differences between corresponding values in the first and third 
columns. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated as: ** = 
1 percent; * = 5 percent. 

The estimated effect size for each measure is calculated as a proportion of the standard deviation of the 
outcome for control group counterparts. 

Cohorts 1, 2, and 3 consist of students in the study who were eighth-graders in the spring of 2005, 2006, 
and 2007, respectively. 

a The "total credits earned toward graduation" measure is the aggregate number of course credits earned 
toward fulfilling the New York State graduation requirements. The credit requirements are as follows: 3 1 
core subject credits, including 8 credits each of English and social studies; 6 credits each of math and 
science; 2 credits of arts; 1 credit of health; and 13 additional credits, including 4 credits of physical 
education, 2 credits of a foreign language, and 7 credits of electives. 

b In order to receive a New York State diploma, students must pass the following core subject Regents 
exams: English Language Arts, Math A, U.S. History, Global History, and one of the science exams 
(Chemistry, Physics, Earth Science, Biology, or Living Environment). The "total Regents exams passed 
toward graduation" measure counts the number of required Regents exams that a student has passed with a 
score of 65 or above. 
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New York City Small Schools of Choice 
Table 3.5 

Estimated Effects of SSC Enrollment on Progress Toward Graduation: 
Third Year of High School, Cohorts 1 and 2 



Target Control 

SSC Group Estimated 

Outcome Enrollees Counterparts Effect 




Effect Size 
(Standard 
Deviation) 


P-Value for 
Estimated 
Effect 


Course credits 














Earned 30 or more credits (%) 


69.5 


62.4 


7.1 


** 




0.000 


Total credits earned toward graduation 3 


32.2 


29.7 


2.4 


** 


0.23 ** 0.000 


Resents exams 














Total Regents exams passed toward graduation 13 


2.7 


2.5 


0.2 


** 


0.11 ** 0.001 


Attendance (%) 














Overall attendance rate 


82.4 


79.4 


3.0 


** 




0.001 


Regular attendance rate (90 percent or higher) 


51.6 


43.4 


8.1 


** 




0.000 


Total number of student observations = 13,297 



SOURCES: MDRC's calculations use High School Application Processing System data from eighth-graders 
in 2004-2005 and 2005-2006, as well as data from New York City Department of Education attendance, 
course credits. Regents exam, and enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: This table presents the estimated effects for students who have follow-up course credits data. 
Appendix A describes how values in the column labeled "Target SSC Enrollees" are estimated. Appendix A 
also describes how values in the column labeled "Estimated Effect" are estimated. Values in the column 
labeled "Control Group Counterparts" are differences between corresponding values in the first and third 
columns. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated as: ** = 

1 percent; * = 5 percent. 

The estimated effect size for each measure is calculated as a proportion of the standard deviation of the 
outcome for control group counterparts. 

Cohorts 1 and 2 consist of students in the study who were eighth-graders in the spring of 2005 and 2006, 
respectively. 

a The "total credits earned toward graduation" measure is the aggregate number of course credits earned 
toward fulfilling the New York State graduation requirements. The credit requirements are as follows: 31 
core subject credits, including 8 credits each of English and social studies; 6 credits each of math and science; 

2 credits of arts; 1 credit of health; and 13 additional credits, including 4 credits of physical education, 2 
credits of a foreign language, and 7 credits of electives. 

b In order to receive a New York State diploma, students must pass the following core subject Regents 
exams: English Language Arts, Math A, U.S. History, Global History, and one of the science exams 
(Chemistry, Physics, Earth Science, Biology, or Living Environment). The "total Regents exams passed 
toward graduation" measure counts the number of required Regents exams that a student has passed with a 
score of 65 or above. 
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exam. The findings also indicate that SSCs increase average attendance during students’ third 
year of high school by 3.0 percentage points and increase the percentage of students who attend 
regularly by 8.1 percentage points. Thus, it appears that SSCs continue to improve student 
progress toward graduation and their engagement in school during the third year in high school. 

Now turn to Table 3.6, which reports findings for a single cohort of students (the ear- 
liest one) over the full four years of high school. This table differs from Tables 3.4 and 3.5, 
which, as described in Chapter 2, present analyses based on the largest available sample for a 
given point in time. Instead, this table follows a single consistent cohort and thus examines 
how enrolling in an SSC affects students’ academic trajectory over time. This trajectory is 
reported in terms of course credits earned and Regents examinations passed. The first mea- 
sure counts only courses that are required for graduation with a New York State diploma and 
the second measure counts only Regents examinations that count toward graduation with a 
New York State diploma. 16 

As can be seen, enrollment in an SSC increases the number of course credits earned to- 
ward graduation in each of the first three years of high school and in the fourth year as well. In 
this way, they “lift” the entire student trajectory toward graduation. On average, target SSC 
enrollees accumulate, over time, 10.8, then 21.6, then 31.9, and finally 39.4 credits toward 
graduation during their first four years of high school. 17 During the same period, control group 
counterparts accumulate 10.2, then 19.8, then 30.2, and then 38.2 credits toward graduation. 
Thus, the average effects of SSCs are stable across the four- year timeframe of this study. 

Findings with respect to the accumulation of Regents examinations over time are less 
clear, most likely because of variation in schools’ policies about when these exams are adminis- 
tered. There is no SSC effect on this outcome in the first year of high school, which is well be- 
fore most Regents exams are typically administered and thus before many students had begun to 
take them. There is still no effect on the accumulation of these examinations by the second year 
of high school. In the third and fourth years of high school, there is a small, positive SSC effect, 
which is not statistically significant and thus may simply reflect random error. 

In summary, then, findings to this point indicate that SSCs consistently improve student 
academic outcomes during their first three years of high school and into their fourth year. The 
next logical question to address is: To what extent do these academic gains translate into in- 
creased rates of high school graduation? 



16 Not all courses or Regents examinations are required for graduation with a New York State diploma. 
Thus, not all courses or Regents examinations count toward graduation requirements. 

17 Recall that not all students who graduate do so at the end of their fourth year of high school. 
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New York City Small Schools of Choice 
Table 3.6 

Estimated Effects of SSC Enrollment Over Time on Total Credits Earned 
and Total Regents Exams Passed Toward Graduation: Cohort 1 





Target 


Control 




Effect Size 


P-Value for 




SSC 


Group Estimated 


(Standard 


Estimated 


Outcome 


Enrollees 


Counterparts 


Effect 


Deviation) 


Effect 


Year 1 of hieh school 

Total credits earned toward graduation 3 


10.8 


10.2 


0.6 ** 


0.16 ** 


0.002 


Total Regents exams passed toward graduation 15 


0.3 


0.3 


0.0 


-0.03 


0.541 


Total number of student observations = 7,891 












Year 2 of hieh school 

Total credits earned toward graduation 3 


21.6 


19.8 


19 ** 


0.24 ** 


0.000 


Total Regents exams passed toward graduation* 5 


1.3 


1.3 


0.0 


0.03 


0.572 


Total number of student observations = 7,492 












Year 3 of hieh school 

Total credits earned toward graduation 3 


31.9 


30.2 


1 7 ** 


0.16 ** 


0.006 


Total Regents exams passed toward graduation* 5 


2.6 


2.5 


0.1 


0.04 


0.375 


Total number of student observations = 6,736 












Year 4 of hieh school 












Total credits earned toward graduation 3 


39.4 


38.2 


1.2 * 


0.13 * 


0.036 


Total Regents exams passed toward graduation* 5 


3.3 


3.2 


0.1 


0.03 


0.513 


Total number of student observations = 5,363 













SOURCES: MDRC's calculations use High School Application Processing System data from eighth-graders in 
2004-2005, as well as data from New York City Department of Education attendance, course credits, Regents 
exam, transactional, and enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: This table presents the estimated effects for students who have follow-up course credits data. Appendix 
A describes how values in the column labeled "Target SSC Enrollees" are estimated. Appendix A also describes 
how values in the column labeled "Estimated Effect" are estimated. Values in the column labeled "Control Group 
Counterparts" are differences between corresponding values in the first and third columns. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated as: ** = 1 
percent; * = 5 percent. 

The estimated effect size for each measure is calculated as a proportion of the standard deviation of the 
outcome for control group counterparts. 

Cohort 1 consists of students in the study who were eighth-graders in the spring of 2005. 

“The "total credits earned toward graduation" measure is the aggregate number of course credits earned toward 
fulfilling the New York State graduation requirements. The credit requirements are as follows: 31 core subject 
credits, including 8 credits each of English and social studies; 6 credits each of math and science; 2 credits of arts; 
1 credit of health; and 13 additional credits, including 4 credits of physical education, 2 credits of a foreign 
language, and 7 credits of electives. 

b In order to receive a New York State diploma, students must pass the following core subject Regents exams: 
English Language Arts, Math A, U.S. History, Global History, and one of the science exams (Chemistry, Physics, 
Earth Science, Biology, or Living Environment). The "total Regents exams passed toward graduation" measure 
counts the number of required Regents exams that a student has passed with a score of 65 or above. 
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Average Effects on High School Graduation During the Fourth Year of 
High School 

Table 3.7 presents estimates of SSC effects on graduation rates as of students’ fourth 
year of high school. (See Box 3.2 for definitions of the outcomes presented in the table.) These 
estimates are presented for the earliest cohort of students to enter the study sample (Cohort 1) 



New York City Small Schools of Choice 
Table 3.7 



Estimated Effects of SSC Enrollment on Graduation: 
Fourth Year of High School, Cohort 1 



Outcome (%) 


Target 

SSC 

Enrollees 


Control 

Group Estimated 
Counterparts Effect 


P-Value for 
Estimated 
Effect 


Graduation 










Graduated from high school 


68.7 


61.9 


6.8 * 


0.013 


Local diploma granted 


24.6 


21.9 


2.8 


0.261 


Regents diploma granted 


39.5 


34.6 


4.9 


0.074 


Advanced Regents diploma granted 


4.4 


5.5 


-1.1 


0.366 


Colleee readiness 










Math A Regents exam score of 75 or above 


22.2 


22.8 


-0.6 


0.787 


English Regents exam score of 75 or above 


34.1 


28.8 


5.3 * 


0.021 


Attendance 










Overall attendance rate 


80.9 


79.0 


1.9 


0.153 


Regular attendance rate (90 percent or higher) 


42.6 


40.1 


2.6 


0.394 


Total number of student observations = 5,363 



SOURCES: MDRC's calculations use High School Application Processing System data from 
eighth-graders in 2004-2005, as well as data from New York City Department of Education 
attendance, course credits, Regents exam, transactional, and enrollment files from the 2005-2006 to 
2008-2009 school years. 

NOTES: This table presents the estimated effects for students who have follow-up course credits 
data. Appendix A describes how values in the column labeled "Target SSC Enrollees" are 
estimated. Appendix A also describes how values in the column labeled "Estimated Effect" are 
estimated. Values in the column labeled "Control Group Counterparts" are differences between 
corresponding values in the first and third columns. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

Cohort 1 consists of students in the study who were eighth-graders in the spring of 2005. 
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New York City Small Schools of Choice 
Box 3.2 

Definitions of Outcomes: Graduation and College Readiness 
Graduation 

• Graduated from high school: This measure includes all students who have received a 
local, Regents, or Advanced Regents high school diploma from the New York State 
public school system. 

• Local diploma: The local diploma represents the most basic graduation requirements in 
the New York State public school system. Over the course of this study, the New York 
State Department of Education phased in new graduation requirements, which will result 
in the elimination of the local diploma except for special education students. To obtain a 
local diploma in the 2009 school year, a student must have completed the 44 credits re- 
quired for graduation, and have passed two of the five core Regents exams with a score 
of 65 or above. The remaining three core Regents exams can be passed with a score of 
55 or above. 

• Regents diploma: The Regents diploma is the standard diploma granted by New York 
City. To obtain a Regents diploma, a student must complete the 44 credits required for 
graduation and pass the five core Regents exams with a score of 65 or above. 

• Advanced Regents diploma: The Advanced Regents diploma is used to signify that a 
student has fulfilled requirements above and beyond what is required for the Regents 
diploma. To obtain an Advanced Regents diploma, a student must complete the 44 
credits required for graduation (including a full six semesters of foreign language) and 
must pass a total of eight Regents exams with a score of 65 or above: the five core Re- 
gents exams, a foreign language exam, a higher-level math exam, and a second 
science exam. 

College Readiness 

• Math A Regents exam score of 75 or above: A binary measure for students who pass 
the Math A Regents exam with a score of 75 or above, thus testing out of developmental 
math in the City University of New York system. 

• English Regents exam score of 75 or above: A binary measure for students who pass 
the English Regents exam with a score of 75 or above, thus testing out of developmental 
English in the City University of New York system. 
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because this is the only cohort for which such information was available. The small size of the 
sample for this analysis (relative to those for earlier high school years) limits the precision of its 
findings. 18 In addition, the limited duration of follow-up for the analysis (four years) minimizes 
the conclusiveness of its findings because many disadvantaged students in large urban districts 
(like those in the present sample) graduate from high school five and six years after they enter. 
Consequently, not all of the graduation “returns” are in for the present cohort. 

The top panel in the table presents estimates of SSC effects on fourth- year graduation 
rates overall and by type of diploma received. These findings indicate that SSCs increase over- 
all graduation rates by 6.8 percentage points, from 61.9 percent for control group counterparts to 
68.7 percent for target SSC enrollees. The 6.8 percentage point increase is equivalent in magni- 
tude to the difference in student graduation rates for schools at the thirty-eighth and fifty-second 
percentiles of the performance distribution for all public high schools in New York City — in 
other words, moving up by approximately 50 schools among the roughly 300 that grant diplo- 
mas. 19 Thus, for the first cohort of students studied and their first four years of high school, the 
evidence indicates that SSC improvements in students’ academic progress and school engage- 
ment during the early years of high school translate into higher rates of on-time graduation. 

Furthermore, findings in the table suggest that a majority of the SSC effect on gradua- 
tion rates reflects an increase in receipt of New York State Regents diplomas. 20 For this type of 
diploma, students must pass a series of Regents examinations with a score of 65 points or above 
and pass all of their required courses. A minority of the SSC effect on graduation rates reflects 
an increase in receipt of local diplomas, which has less stringent standards for scores on Regents 
examinations. This type of a diploma will be phased out by New York State for the high school 
class graduating in 20 10. 21 

The middle panel of Table 3.7 reports estimates of SSC effects on the percentage of 
students who pass the Math A or English Regents examinations with scores of 75 points or 
higher. This is a higher standard than is necessary to pass each examination (which is 65 points) 
and represents the threshold for exempting incoming students at the City University of New 
York from remedial courses in these subjects. These findings are included to begin to explore 



1 bindings for student subgroups for fourth -year outcomes are presented in Appendix Table F.l. Because 
of the small sample size, few findings are statistically significant (all for those subgroups that are themselves 
relatively large). Thus, most of the effects reported in the table could be understood as having occurred by 
chance. 

19 School performance percentiles are based on publicly available 2007-2008 DOE Report Card data. 

~°The effect on the percentage of students receiving a NYS Regents diploma (p-value = 0.074) approaches 
the level of statistical significance at the 0.05 level. 

21 In the 2007-2008 school year, the citywide overall graduation rate was 60.7 percent. That included 40.9 
percent of the students earning a Regents diploma, which is considered by New York State to be the standard 
diploma, and 15.5 percent of the students earning a local diploma, which New York State was phasing out over 
time. 
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SSC effects on students’ college readiness. There is no observed effect in math and a 5.3 per- 
centage point increase in English. 

Lastly, the bottom panel of Table 3.7 suggests that during the fourth year of high 
school, SSCs continue to improve students’ regular attendance — and, thus, students’ school 
engagement — although these findings are not statistically significant. 

In closing, the current analyses reveal that SSCs exhibited promising effects on the first 
cohort of target SSC enrollees that were sustained throughout four years of high school, leading 
to improved graduation rates for this group. While the findings are very promising, they 
represent only one cohort of students, and the follow-up period does not capture all potential 
SSC graduation effects even for this cohort (because some students take more than four years to 
graduate). When data for future cohorts and follow-up years become available, it will be impor- 
tant to learn whether these early positive findings are sustained and replicated. 
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Chapter 4 

Conclusions and Next Steps 



This report emerges at a moment when policymakers, practitioners, and researchers 
have identified the high school years as the point of greatest need within the education pipeline. 
The rationale for this collective focus is clear: far too many students drop out of high school, 
and the consequences of entering adult life without a high school diploma are increasingly 
grave. Amid a national call for change and a dearth of effective responses, the findings pre- 
sented in this report provide encouraging and reliable evidence that: 

• In roughly six years’ time it is possible to create a large system of small high 
schools of choice (SSCs) that markedly improve graduation prospects for 
many disadvantaged students who choose to attend these schools. 

• Positive effects on students’ progress toward high school graduation first 
become apparent during the ninth grade; these effects are sustained during 
the next two years, when most students are in grades 10 and 11; by the 
end of four years of high school, these effects culminate in higher rates of 
graduation. 

• These effects are experienced by a broad range of students who differ in 
tenns of their demographic characteristics, economic circumstances, and 
academic preparation. Of particular note is that these effects are experienced 
by male high school students of color, whose educational prospects have 
been historically difficult to improve. 

This chapter contextualizes these findings and identifies their key implications for fu- 
ture policy, practice, and knowledge building. 



Interpreting the Findings 

The effects of small high schools of choice described in this report should be under- 
stood through three important lenses: (1) their scale, (2) the comparison they represent, and (3) 
the group of highly disadvantaged students for whom they occurred. 

Effecting Change at Scale 

As described in the preceding chapters, the present study estimates the effects of 105 
small schools of choice on student achievement. At capacity, those schools will serve over 
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45,000 students. 1 That is roughly equivalent to the entire high school population of Houston, 
which is the seventh largest school district in the country. 2 Readers should understand the 
magnitude of the present report’s findings in that context — imagine, for a school district the 
size of Houston, increasing the percentage of ninth-graders who are eligible for on-time promo- 
tion by 10.8 percentage points, 2 the percentage of black males on track to graduate by the end of 
ninth grade by 8.5 percentage points, or the percentage of high school graduates by 6.8 percen- 
tage points. Given the scale of the SSC initiative, even seemingly minor gains can be under- 
stood as affecting thousands of high school students over time. In fact, the 6.8 percentage point 
increase in four-year graduation rates is roughly one-third the size of the gap in New York 
City’s graduation rates between white students and students of color. 

Additionally, because the reported effects of SSCs are not the product of a small, tar- 
geted intervention but rather of a large system of small schools, the effects can be understood 
as reflecting the mean performance of a model implemented at scale. Reported effects are not 
the product of the best or most popular of the SSCs but of 105 schools on average. Although 
it is beyond the scope of this study to analyze variation in effects based on school characteris- 
tics, a review of the data suggests that the average effects reported are produced by a mix of 
higher- and lower-performing schools. In other words, the findings represent a real-world test 
of an intervention launched at the scale of a large-sized urban school district. 

Understanding What Distinguishes Small Schools of Choice from 

Other Schools 

As described in the first three chapters of this report, the reported effects on student 
achievement are driven by the special school experience of students enrolled in SSCs compared 
with the experiences of their counterparts who enrolled in a wide range of other public high 
schools in New York City. In interpreting the findings, it is important to review the essential 
elements of both of those experiences. 

Students enrolled in SSCs did not just attend schools that were small. SSC enrollees at- 
tended schools that were purposefully organized around smaller, personalized units of adults 
and students, where students had a better chance of being known and noticed, and teachers had 



SSCs are intended to serve 108 students per grade, for a capacity of approximately 430 students through- 
out grades 9-12. 

2 See Houston Independent School District (2009). 

3 See Table 3.2 and note that the measure listed as “Earned 10 or more credits” in the first year of high 
school serves as a proxy for on-time promotion to the tenth grade. 
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a better chance of knowing enough about their charges to provide appropriate academic and 
socioemotional supports. 4 

Similarly, SSCs were not only new but were mission-driven. Their recent establishment 
via a demanding authorization process, which resulted in more schools being denied than 
approved, 5 required that a prospective school leadership team articulate an educational philoso- 
phy and demonstrate how it would motivate teachers, community members, and partner 
organizations around it. The purposeful process employed to create new schools required new 
school leadership to articulate and justify a viable improvement strategy. And the district’s 
commitment to acting as a steward for new schools through their start-up period generated a set 
of supports and protections as these schools got up and running. This scenario stands in contrast 
to school closing and opening processes in some other districts where schools close and later 
reopen as a set of small schools or learning communities, but with the same leadership, teachers, 
and students. 6 

Finally, SSCs benefited from an influx of external ideas, talent, and resources. A 2006 
MDRC report synthesizing five overarching lessons drawn from studies of whole school reform 
models notes that significant amounts of “time, energy, and know-how” are required to stimu- 
late large-scale change. 7 As noted elsewhere in this report, SSCs seemingly had a supply of all 
three: a protected start-up period; a newer teaching corps; and the support of independent, 
external, nonprofit intermediary organizations that had experience starting and operating small 
schools. 



The other set of school experiences that must be understood in order to interpret SSC 
enrollment effects are those of the study’s control group counterparts, which serve as the 
comparison to the school experience described above. This “counterfactual” school experience 
represents the environment that target SSC enrollees would have experienced had they not been 
randomly assigned to an SSC, and it is the contrast between the two states that drives the 
reported effects. 

Importantly, this report does not contrast the experience of target SSC enrollees with 
that of the students who attended the 23 failing high schools that the SSCs replaced. Because 
the closing schools ceased to admit freshmen as new SSCs came into being, the two options 
were not simultaneously available to incoming students. Because the present report could not 
contrast SSC effects with this original counterfactual, it likely understates the effects of SSCs 



4 Appendix H contains a sample of the DOE’s New School Application, which details many of the spe- 
cialized supports that new small schools were intended to provide to their students. 

5 Intemal memo. New Visions for Public Schools (2005). 

6 See Evan et al. (2006). 

7 Quint (2006), p. 53. 
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that might have been attained if the SSCs could have been reliably compared with their antece- 
dents. 



Instead, the counterfactual in this study comprises not simply one school type but rather 
a broad range of high schools that differ along several dimensions. While the schools attended 
by control group counterparts are larger and older on average, they are not universally so. For 
example, while the schools attended by control group counterparts had 506 more students in 
their average ninth-grade class than did schools attended by target SSC enrollees, 12.5 percent 
of the control group counterparts attended comprehensive schools that, although they may have 
had large incoming classes, had been reorganized into small learning communities, thus 
providing a “smaller” experience. While control group counterparts attended schools that were 
generally older, a full one-fourth of them attended schools that, like those attended by target 
SSC enrollees, had been created or restructured since 2002. 

The diversity of high school options experienced by control group counterparts under- 
scores two important points. First, this report does not provide a straightforward comparison of 
small schools to large schools or new schools to old ones, but rather a comparison of SSCs to a 
range of contemporaneous alternatives. Second, the comparison is drawn at a point when the 
system was undergoing a wholesale transformation. Neither the SSC “treatment” nor the 
counterfactual experience had yet reached a “steady state.” 

A Strategy to Serve Disadvantaged Students 

SSCs were intended to be a viable and accessible option for the district’s most disad- 
vantaged students, and over the course of the study period, they served a population that 
almost exclusively comprised low-income students of color. 8 The fact that SSCs targeted and 
served this population gives the reported findings even greater policy significance, as it is 
precisely these sorts of students who find themselves at the bottom end of the country’s 
persistent achievement gap, and who are least likely to graduate from high school on time, if 
at all. For certain subgroups, the statistics are particularly alarming. For example, a recent 
analysis by researchers at New York University revealed that among the cohort that began 
high school in New York City in 2001, only 44 percent of black and Latino males had 
graduated after six years. 9 

Given the small number, and typically limited scale, of interventions that have demon- 
strated success serving low-income students of color, the effects produced by SSCs mark these 

8 Table 2.3 characterizes first-time ninth-graders who were enrolled in the study SSCs. In summary, 93.6 
percent of those students were black or Latino, and 83.8 percent came from low-income households, as 
indicated by their qualification for free or reduced price lunch. 

9 See Meade, Gaytan, Fergus, and Noguera (2009). 
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schools as a model that can serve such students en masse. Furthermore, the robust positive SSC 
enrollment effects for many different types of students, including young men of color, represent 
findings that are highly encouraging. 



Implications of the Findings 

This section considers how the preceding findings can be applied to current educational 
policy and practice, and identifies the open questions raised by the findings that should be 
addressed through future research. 

For Policy and Practice 

The timeliness of the present study hinges largely on the country’s current focus on 
high school reform, which can be seen at the federal, state, and local levels on the part of 
legislators and practitioners alike. As part of the 2009 stimulus package, the federal government 
committed billions of dollars toward education reform, prompting states and districts to develop 
action plans for the use of formula-based dollars and comprehensive proposals for the competi- 
tive funding streams that were established. At the outset of the 2010 calendar year, Congress 
began debating the reauthorization of the Elementary and Secondary Education Act (last 
reauthorized as No Child Left Behind), which will govern the next generation of federal 
education policy. And at the district level, as state and district education budgets fall victim to 
the current economic crisis, superintendents are being compelled to do more with less, targeting 
limited resources at those points believed to give them the greatest leverage. 

With respect to high schools in particular, much of the national discussion focuses on 
three areas where the education community has struggled to demonstrate success: (1) improving 
the academic outcomes of the most disadvantaged students, particularly with respect to high 
school graduation and college readiness; (2) identifying turnaround strategies for historically 
underperforming schools; and (3) implementing effective interventions at scale. The present 
study sits at the nexus of all three themes, and its findings demonstrate that, in a short period of 
time, an effective model can be implemented at scale and can improve the academic trajectories 
of large numbers of traditionally underserved students. 

The reforms implemented in New York City should be considered as a package of inte- 
grated, reinforcing strategies. The effects are not simply the result of closing low-performing 
schools or of creating SSCs, but rather a purposeful marriage of the two strategies supported by 
the implementation of several enabling reforms (such as the introduction of a districtwide 
choice process). Decision-makers interested in replicating the district’s strategy should devote 
as much attention to how these reforms were operationalized as they do to what was conceptua- 
lized. Closing the failing schools would likely not have been singularly effective without the 
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intentional creation of a range of alternative options to pick up the slack. Similarly, the creation 
of new schools would likely not have gained the traction it did without the introduction of a 
districtwide choice process that compelled previously disenfranchised communities to explore 
their high school options and exercise choice. Thus, while this study provides compelling 
evidence in support of a particular small school model, that model cannot be understood as 
existing in isolation but rather as an integral component of a comprehensive and coordinated set 
of district reforms. 

For Knowledge Building 

The findings presented in this report should provide much-needed encouragement to 
educational policymakers, practitioners, and researchers by demonstrating that it is possible to 
transform a major system of high schools in ways that benefit large numbers of disadvantaged 
students. To build on these findings, future research should focus on what is needed to facilitate 
educational improvement on a broad scale with a high probability of success. 

To address these further questions, the present research should be advanced on three 
fronts: (1) quantitative and qualitative analyses of what it is about the nature of new small high 
schools of choice in New York City that make them effective for disadvantaged students; (2) 
statistical analyses of SSC effects on graduation rates for additional student cohorts and after 
students’ first four years of high school, and the extent to which high school effects translate 
into gains on future educational outcomes (such as college-going and success) and economic 
outcomes (employment and earnings); and (3) quantitative analyses of whether effects persist as 
the school system of which they are a part evolves. 

Identifying the Locus of the Impacts 

The present analysis focuses mainly on average effects of SSCs and, to a limited extent, 
on how these effects vary across different types of students. But much more should be done to 
explore in detail the magnitude and nature of variation in effects across different types of SSCs. 
In other words, it is essential to begin to unpack the “black box” of the current preliminary 
analysis in order to develop a more comprehensive understanding of “what works best for 
whom, when, where, and why.” 

The more robust the effects are for different types of SSCs, the more confidence one 
can have that SSCs created under the conditions experienced by those in New York will have 
similar effects in other places, at other times, and for other students. The less robust these effects 
are, the more uncertainty there will be about the likely effectiveness of any given future attempt 
to replicate SSCs. 
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This research should begin by using existing data to study how variation in SSC effec- 
tiveness is related to characteristics of SSCs and the partnerships through which they were 
created and operated. For example, statistical analysis could explore the extent to which 
variation in SSC effectiveness is related to the SSCs’ educational approach by categorizing 
schools as being more explicitly focused on a traditional, liberal arts curriculum or on experien- 
tial elements such as work-based or service learning. Similarly, analyses might explore whether 
the SSC effectiveness varies based on schools’ model of intermediary support — that is, the 
SSCs’ affiliation with an intermediary that prioritized leadership development, teacher and 
student services, or community relations. Finally, future analyses could investigate whether SSC 
effectiveness is related to the experience and turnover of teachers and principals, characteristics 
that have been identified in the literature as being correlated with student outcomes . 10 

A second important line of research that can begin now with analyses of existing data, 
but will require additional data in the future to complete, is a systematic analysis of how the 
effectiveness of SSCs evolves as they mature over time. This future research would build 
directly on past research indicating that three to five years are required for a major school 
reform to become effective . 11 This research would also build directly on the expectations of 
funders and creators of New York’s SSCs who provided them with special protection and 
support during their first two years of operation. This assistance was provided in anticipation 
that the first two years of an SSC’s operation would be especially difficult. 

Preliminary analyses conducted by the authors suggest that there was variation in the ef- 
fects produced by study SSCs that were in their first two years of operation compared with 
schools that had been in operation for three or more years. However, it is not yet possible to 
identify whether the effectiveness of SSCs changes as they mature over time. This problem 
arises because many new SSCs were created each year during the present analysis period. Thus, 
the group of SSCs that were in their first two years of operation is largely different from the 
group of SSCs that had been in operation for three years or more. With four cohorts of data and 
the development of a more sophisticated statistical model, it should be possible to begin to shed 
further light on this issue . 12 And with future data for additional cohorts of incoming ninth- 
graders, it should be possible for this analysis to be conclusive. 



10 See Weinstein et al. (2009). 

1 Borman. Hughes, Overman, and Brown (2003). 

12 For this purpose, it will be helpful to specify differences in effects across SSCs and over their develop- 
mental trajectories as random effects, which will require an extension of the instrumental variables analysis 
framework used for the present report. 
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Understanding Long-Term Effects on Student Achievement 

The findings presented in this report are particularly notable because of the effects on 
graduation rates, which have been absent from many other studies of whole school reform 
models, even those that showed strong effects in earlier years. Thus, it will be important to 
analyze graduation rates and longer-term effects in greater detail using additional data that can 
be obtained in the future. 

One such analysis will include the variation in graduation rates across subgroups of 
students. That analysis was quite limited for the present report because data on graduation rates 
are available only for one annual cohort of students. But as data for additional cohorts become 
available, much more can be learned about the robustness of the “bottom-line effect” of SSCs. 

A second line of research is analysis of SSC effects on graduation rates that account for 
the fact that many disadvantaged students who graduate from high school do so only after five or 
six years. Such an analysis would extend current findings on four-year graduation rates in a way 
that provides a more complete picture of the effects of SSCs on students’ academic success. 

A third area of analysis aims to identify some of the key leverage points associated with 
increased graduation rates. That analysis will investigate whether certain course- and Regents- 
taking patterns are correlated with graduation effects. 

Another extension of the present research is analysis of the extent to which SSC effects 
on high school graduation translate into positive effects on future educational outcomes, such as 
college or community college enrollment, persistence, and graduation, and future economic 
outcomes, such as employment and earnings. This research could follow existing and future 
cohorts of students through administrative records on college-going obtained from the National 
Student Clearinghouse and New York City’s public university system, and on future earnings 
and employment data obtained from state unemployment insurance records. Given the promis- 
ing findings to date with respect to SSC effects on four-year high school graduation rates, it is 
especially important that the potential longer-term effects of this initiative be examined. 

Understanding Whether Effects Persist as the System Moves Toward 

“Steady State” 

A final area of analysis that can begin with existing data but requires additional data in 
the future is an exploration of how the effectiveness of SSCs as a group changes over time as 
their school system undergoes major changes. As noted earlier, the four annual cohorts of 
incoming ninth-grade students in the present analysis all experienced similar effects of enrolling 
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in an SSC. This is surprising given the dramatic changes that were occurring concurrently 
throughout the New York City school system during this period . 13 

Some of the literature describing the small schools created in New York City suggests 
that the schools that had been created as part of earlier waves of the district’s reforms were 
better positioned to operate effectively — competing against a relatively small group of other 
new schools for motivated, entrepreneurial educators and for an abundance of foundation and 
district resources . 14 This line of reasoning suggests that over time, as many more new small 
schools came into existence, they were increasingly forced to compete for a shrinking group of 
top principals and teachers, thereby diminishing their likely chances of success. Additionally, if 
the New York City Department of Education is correct in hypothesizing that systemic changes 
related to the principles of accountability, leadership, and empowerment would cause all boats 
to rise, there is the potential for a diminished contrast over time between the school experience 
offered by the front-runner SSCs and those offered by the remainder of the system’s schools. 



Conclusion 

This study provides important findings about an unusually promising educational initia- 
tive — reliably demonstrating for the first time that transformation in a large urban school 
system at scale, serving disadvantaged students, is possible in a relatively short time period. But 
the true value of this study lies in its potential contribution to improving future education policy 
and practice, because while these impacts were strong, they are not enough. There remains 
much to learn and do in order to build on the process described here and to make future reform 
possible, so that concrete results can be achieved — particularly in terms of narrowing the 
educational achievement gap between historically underserved students and their better-off 
peers. Given the robust positive findings observed to date, however, there is good reason to 
believe that turning around our lowest-performing schools and thereby improving the academic 
prospects of this country’s most disadvantaged children is a realistic goal. 



13 Once again, however, it was not possible to reliably separate differences among SSCs (because new 
SSCs were opening every year) from changes in the same SSCs over time. 

14 See Hemphill and Nauer (2009); Foley, Klinge, and Reisner (2007). 
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Appendix A 

More About the Study’s Research Design and Analysis 




For readers who are interested in further details about the present study’s research de- 
sign and analysis, this appendix describes: 

• How New York City’s High School Application Processing System 
(HSAPS) creates randomized SSC lotteries 

• How lotteries for small schools of choice (SSCs) are used to estimate effects 
of winning them 

• How these estimates are converted into estimated effects of enrolling in an 
SSC 

• How the plausibility of assumptions needed for estimates is assessed 

• How estimates are reported 



How HSAPS Creates SSC Lotteries 

This section briefly describes how HSAPS creates the statistical equivalent of a ran- 
domized lottery for each SSC that is oversubscribed in a given year. An SSC that is not over- 
subscribed in a particular year does not have a lottery for that year. Although HSAPS assign- 
ment does not operate literally as described below, the results of HSAPS are the same as those 
produced by the assignment rules and statistical properties described below. 

Recall that each student provides a list of up to 12 high schools that he or she would 
like to attend, in order of preference. Also recall that each SSC gives to HSAPS its student 
priorities based on their geographic proximity and whether they are “known” to the SCC (by 
having contacted it, having visited it, or having met with one of its representatives). As noted 
in Chapter 2, geographically most SSCs distinguish only between residents of their borough 
and all other New York City residents. 1 Within these categories, the highest priority is given 
to students who both satisfy the school’s geographic preference and are known to the school, 
whereas the lowest priority is given to students who do not carry a geographic preference and 
are not known to the school. 

HSAPS assigns students to schools based on the inputs from both parties. The key to 
this process, and its ability to create randomized SSC lotteries, is the fact that the order in which 
students are chosen for school assignment is random. Using a computer algorithm, HSAPS 
chooses randomly the first student to be assigned, the second student to be assigned, the third 
student to be assigned, and so forth, all the way to the last of roughly 80,000 students to be as- 
signed. Students who are chosen for assignment very early in this process are the most likely to 



'A minority of SSCs have one or more additional geographic priorities, such as priority to residents 
of a nearby catchment area. 



69 




receive their first choice because at this point no schools are filled to capacity. (However, high- 
er-priority students who are assigned later in the process can “bump” students who were as- 
signed earlier in the process but had lower priority for that school.) At some point in the as- 
signment process, however, schools begin to fill up and their student priorities start to take 
effect. For example, if Student A’s first-choice school is already filled by students with equal or 
higher priority, Student A cannot be assigned to that school. Instead, HSAPS assigns Student A 
to the next-preferred school on the student’s list with available slots. (Consider what happens 
each time the algorithm tries to assign a student to a school that is full. If the current student has 
higher priority for that school than do one or more students who are assigned there already, the 
current student bumps the last student assigned who had the lowest priority. The student who is 
bumped is then assigned to his or her next-preferred school with available slots, following the 
same rules.) This process continues until HSAPS works through the entire randomly ordered 
cohort of incoming students. 

Figures A. 1 and A.2 provide heuristic descriptions of what the results of HSAPS as- 
signment are like for a hypothetical student and a hypothetical SSC. Figure A. 1 illustrates what 
these results are like for a student who is assigned to his third-choice school, which happens to be 
an SSC. In this scenario, the student is not assigned to his first-choice school (an SSC) because 
his randomly assigned order within HSAPS was “too late” (after other students with the same 
priority had filled up that school). As explained below, this means that the student “lost a lottery” 
for the SSC and is a member of its control group. In the scenario shown, the student also does not 
get assigned to his second-choice school, which is not an SSC. The reason he does not get as- 
signed to this school is that it ultimately is filled to capacity with students who have higher priori- 
ty. Thus, the student is not in a lottery for his second-choice school. Finally, the student is as- 
signed by HSAPS to his t hird-choice school, which happens to be another SSC. Because this 
SSC is ultimately oversubscribed by students with the same priority as that of the current student, 
in effect he is a lottery winner, or a member of the “treatment group” for that SSC. 

In summary, then, HSAPS assignment for the hypothetical student involves ( 1 ) losing a 
lottery for the first-choice school (an SSC) and thereby becoming a control group member for it; 
(2) not being assigned to the second-choice school (which is not an SSC) because it was filled 
by other students who had higher priority for the school based on whatever method that school 
used to select students; and (3) winning a lottery for the third-choice school (another SSC) and 
thereby being assigned to it. Students and their families, however, see only the rank-ordered list 
of schools they submit to HSAPS and the school to which HSAPS assigns the student; the lot- 
tery itself is invisible to them. 

Figure A.2 illustrates the results of HSAPS assignment for a hypothetical SSC that can 
accommodate up to 120 incoming ninth-graders. These results represent all of the students that 
HSAPS attempted to assign to the SSC and can only be finalized after the HSAPS assignment 
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New York City Small Schools of Choice 
Appendix Figure A.l 

HSAPS Assignment Process for a Hypothetical Student 




process has run to completion. First note that HSAPS will attempt to assign to this SSC all stu- 
dents who list it as a choice and have not been assigned to one of the choices they prefer more. 
For example, students who list the SSC as their third choice and are not placed into their first or 
second choice are considered for assignment to this SSC. However, students who list the SSC 
as their third choice but are placed into their first choice or second choice are not considered for 
assignment to this SSC because they already are assigned elsewhere. 
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New York City Small Schools of Choice 
Appendix Figure A.2 

HSAPS Assignment Process for a Hypothetical SSC 
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= Students assigned to program 
= Students not assigned to program 

= Cell containing a lottery 



In Figure A.2, 360 students have listed the hypothetical SSC as a choice and have not 
yet been placed elsewhere. Thus, this SSC has 360 “potential assignees” for its 120 places. 
Eighty of those potential assignees are both known to the SSC and they reside in its borough, so 
they have first priority for this school (Priority Cell 1); 160 are not known to the school but they 
reside in the school’s borough, so they have second priority (Priority Cell 2); 50 are known to 
the school but do not reside in its borough, and so have third priority (Priority Cell 3); and 70 
are neither known to the school nor reside in its borough, so they have fourth, or last, priority 
(Priority Cell 4). Given the SSC’s capacity of 120 available ninth-grade spaces, it can accept all 
80 students from Priority Cell 1 plus the first 40 students assigned by HSAPS (in random order) 
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from Priority Cell 2. It cannot accept the last 120 students assigned from Priority Cell 2 or any 
students from Priority Cells 3 and 4. 2 

Because Priority Cell 2 is oversubscribed, it represents the equivalent of a randomized 
lottery for assignment to the SSC. The 160 students in this cell are effectively “lottery partici- 
pants.” The first 40 of these participants whom HSAPS selects randomly are lottery winners 
and are assigned to the SSC, thereby becoming members of its treatment group. The last 120 
participants whom HSAPS selects randomly are lottery losers and are not assigned to the SSC, 
thereby becoming members of its control group. 

No lottery exists for Priority Cell 1 because all of its potential assignees can be accom- 
modated by the SSC; no lottery exists for Priority Cells 3 and 4 because none of their potential 
assignees can be accommodated by the SSC. This is what HSAPS assignment is like for an 
oversubscribed SSC. However, all that the SSC sees is the list of known students that it submits 
to HSAPS and the list of students that HSAPS assigns to it. 3 

Most lotteries are for Priority Cells 1 and 2. Very few are for Priority Cells 3 and 4. As 
stated earlier, HSAPS lotteries are not held for SSCs that are not oversubscribed in a given year 
and they are not represented in the present analysis for that year. 



How Effects of Winning an SSC Lottery Are Estimated 

This section outlines the basic approach used to estimate effects of winning an SSC 
lottery, describes two necessary extensions to this approach, and presents the resulting statis- 
tical model. 

Basic Approach 

Randomization ensures that observed mean outcomes for control group members are 
valid estimates of what the outcomes would have been for SSC lottery winners had they lost 
their lottery — known as “counterfactual” outcomes for SSC lottery winners. Observed differ- 
ences between mean outcomes for SSC lottery winners and outcomes for control group mem- 
bers, therefore, are valid estimates of the average effects of winning (as opposed to losing) an 

2 In this heuristic, the priority sequence of the SSC’s cells is defined as (1) geographic preference and 
known, (2) geographic preference and not known, (3) no geographic preference and known, and (4) no 
geographic preference and not known. In other words, geographic preference trumps whether a student is 
known to the school (note the sequence of Priority Cells 2 and 3). This may not have been the case in all 
years of HSAPS, and may change in future years to reflect DOE policy. However, the sequence of the 
cells does not affect the randomized results of the SSC lottery as lotteries take place in just one cell per 
school, among students who all share the same priority combination. 

3 HSAPS determines the geographic priority of each student for each school based on the addresses of 
the student and the school and the geographic priority categories of the school. 
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SSC lottery. The basic approach for the present analysis is thus to estimate for each SSC lottery 
differences in mean outcomes for winners and control group members and to average the results 
across lotteries. Because of technical issues created by HSAPS assignment, the following two 
extensions of the basic approach were required. 

Extension 1: Accounting for Clustering Produced by Students Who Participate 
in More Than One SSC Lottery 

Recall that the hypothetical student in the HSAPS assignment process described above 
(Figure A.l) lost an SSC lottery for his first-choice school and won an SSC lottery for his third- 
choice school. Thus, he participated in two SSC lotteries. It is actually possible for a student to 
participate in more than two SSC lotteries, for example, by losing two or more lotteries before 
winning one or by losing more than two lotteries. Consequently, 24 percent of students in the 
present sample are participants in more than one SSC lottery. 4 This implies that they represent 
more than one “student observation” in the analysis. For example, the hypothetical student in 
the discussion above is represented as an SSC lottery winner (or student observation in the 
treatment group) for his t hird-choice school and as an SSC lottery loser (or student observation 
in the control group) for his first-choice SSC. This student, therefore, is represented by two ob- 
servations for any given estimate of the average effect of winning an SSC lottery. However, 
because of the way these estimates are computed (by estimating the effect of winning each SSC 
lottery and averaging those estimates across lotteries), each individual student carries an appro- 
priate weight in the overall result. This is directly analogous to how “matching with replace- 
ment” is used to estimate intervention effects. 5 Thus, no additional steps are required to account 
for multiple lottery participation in order to compute unbiased estimates of the average effects 
of winning an SSC lottery. 

It is necessary, however, to account for multiple lottery participation in order to obtain 
appropriate estimates of standard errors, because multiple observations for a given student are 
correlated with each other. Indeed, they are perfectly correlated with each other and thus have 
an intraclass correlation of l.O. 6 To properly account for this fact, all observations on a given 
outcome for a given student can be specified as being “clustered” by that student. This can be 
accomplished using the standard clustering option that exists for most current statistical soft- 



4 Specifically, 75.6 percent of all students in the sample participated in one SSC lottery, 19.3 percent 
participated in two SSC lotteries, 4.0 percent participated in three SSC lotteries, and 1.2 percent partici- 
pated in four or more SSC lotteries. 

5 See Austin (2008). 

6 The intraclass correlation for student clusters of observations equals a value of 1.0 because out- 
comes do not vary across observations for a given student. 
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ware. 7 In this way, unbiased estimates of standard errors were obtained for results in the 
present report. 

Adding this clustering option to estimates of effects of winning an SSC lottery on the 
likelihood of making adequate progress toward graduation in ninth grade for the full study sam- 
ple of four entering cohorts increases the resulting standard error by about 6 percent of its value 
without accounting for clustering. Thus, although in principle students’ participation in multiple 
lotteries could have a major effect on the precision of the present analysis, in practice it has very 
little effect. 

Extension 2: Controlling for Variation in the Probability of Students’ Prior 
Assignment Created by Participation in Lotteries for Preferred (or Prior-Choice) 
Schools 

There is one and only one potential threat to the randomization produced by an SSC lot- 
tery. This potential threat derives from variation in students’ ex ante probability of assignment 
to a preferred (or prior-choice) school based on their random HSAPS assignment order. The 
probability of prior assignment for an SSC lottery participant is the probability that HSAPS 
could have by chance assigned him or her “early enough” to win a lottery for a prior-choice 
school. As explained below, variation in this probability can in theory cause SSC lotteries to 
produce lottery winners and control group members who differ at baseline and thus have differ- 
ent mean counterfactual outcomes. In practice, however, this does not appear to occur. 

For three-fourths of SSC lottery participants there is no chance that HSAPS assignment 
order can enable them to win a lottery for a prior-choice school. This is either because they had 
no prior-choice school (the current lottery was for their first choice) or because lotteries had 
nothing to do with why they did not get a prior choice — that is, they were preempted by stu- 
dents with higher priority or students who satisfied admissions criteria (for schools that were not 
SSCs). For this majority of lottery participants, the probability of prior assignment equals 0 and 
there is no threat to randomization. 

For one-fourth of SSC lottery participants, there is some chance that HSAPS assign- 
ment order could enable them to win a lottery for a prior-choice school. This is because they 
were in a lottery for one or more prior school choices. 8 For participants in one prior lottery, the 
probability of prior assignment is approximately the proportion of participants who won that 



7 White’s cluster-robust standard errors are used to account for student-level clustering, which occurs be- 
cause some students appear in multiple lotteries and are therefore included in the analysis more than once. 

8 Some high schools that are not SSCs are oversubscribed in some years and thus have an HSAPS lot- 
tery. Their lotteries are accounted for in the present analysis. 
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lottery. 9 For example, if one out of three participants won this prior lottery, the probability of 
prior assignment equals about 33 percent. For participants in more than one prior lottery, the 
probability of prior assignment equals approximately the proportion of lottery participants who 
won their least competitive lottery. This is because they only had to win it to be assigned by 
HSAPS to a prior-choice school; they did not also have to win a more competitive lottery. For 
example, if a student were in two prior lotteries, with a one-out-of-three chance of winning one 
lottery and a one-out-of-four chance of winning the other lottery, the probability of prior as- 
signment would equal one-third (the probability of winning the less competitive lottery). 

For this minority of lottery participants, the probability of prior assignment can vary 
from just above 0 to just below 1, and this variation can pose a threat to randomization. For 
example, assume that academically weak students choose unpopular schools whose lotteries are 
weakly competitive (9 out of 10 participants win). Assume that all other students choose more 
popular schools with more competitive lotteries (1 out of 10 participants wins). Because weak 
students are in weakly competitive prior lotteries, those who lose these lotteries must have been 
assigned especially “late” by HSAPS (otherwise they would have won). In contrast, because 
other students are in more competitive prior lotteries, students who lose these lotteries can have 
a mix of “early” and “late” HSAPS assignments. 

Recall that among students who participate in prior lotteries, only those who lose are 
available to participate in a subsequent lottery; prior-lottery winners get assigned to the school 
for their prior lottery. In this way, the population of participants in subsequent lotteries is “pre- 
screened” on their HSAPS assignment order. Consequently, this order may not be fully ran- 
domized across the remaining subpopulation of participants in subsequent lotteries. 

In the present hypothetical example, the HSAPS assignment orders of weak students in 
a current lottery are later than those of other students in the current lottery because of their pre- 
screening by prior lotteries. Weak students, therefore, will be more likely than others to lose 
the current lottery and be in its control group. Mean future outcomes for control group mem- 
bers in the current lottery thus will understate counterfactual outcomes for lottery winners. 



9 Exact values of this probability also depend on the degree to which other prior choices for lottery 
participants are oversubscribed. Thus, it is possible that lotteries with the same ratio of winners to partici- 
pants could involve students with different distributions of HSAPS assignment orders. Hence, a student’s 
prior probability of winning these lotteries might differ. In theory, this possibility could be controlled for 
(for the one-fourth of lottery participants who were in prior lotteries) by grouping students into “risk sets” 
defined by exact permutations of their prior choices (as done by Hastings, Kane, and Staiger, 2006, and 
Abdulkadiroglu et al., 2009). In practice, this is not feasible for the present analysis because it would re- 
quire tens of thousands of risk sets, most of which would contain a single student and thus not represent a 
lottery. Furthermore, as demonstrated below, this approach is not necessary for the present analysis be- 
cause without any statistical adjustments the baseline characteristics of SSC lottery winners and control 
group members are virtually identical. Thus, randomization appears not to be compromised by this theo- 
retically possible phenomenon. 
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Consequently, future outcome differences for the two groups will overstate the effect of win- 
ning the SSC lottery. 

The preceding illustration is, of course, only a theoretical possibility. Its practical im- 
portance depends on the strength of whatever systematic relationships exist among students’ 
counterfactual future outcomes, their prior school choices, and the degree to which those prior 
choices are oversubscribed. There is no compelling theoretical reason to expect these relation- 
ships (especially their overall product) to be strong given the large amount of random variation 
that exists for each factor. However, whether these relationships are strong enough to be prob- 
lematic remains an empirical question. 

The most direct empirical evidence on this question is generated by the comparison of 
mean baseline characteristics for SSC lottery winners and control group members (presented in 
Table 2.1 of Chapter 2 and repeated here in Appendix Table A.l). These findings clearly indi- 
cate that the two groups are virtually identical at baseline on a wide range of characteristics that 
typically predict future academic success, including students’ scores on standardized tests of 
reading and math in eighth grade. 

If variation in the prior probability of assignment were sufficiently problematic to create 
a systematic difference in counterfactual outcomes for SSC lottery winners and control group 
members, then a reflection of this difference should be visible in their baseline characteristics, 
especially in their eighth-grade test scores. There is no sign of this difference, however. 

Appendix Table A.2 (see page 80) provides further evidence that no such difference ex- 
ists by comparing SSC lottery winners and control group members in terms of their number of 
preferred choices not received prior to the current lottery in which they are participating. 10 These 
findings indicate that 41.3 percent of observations for both lottery winners and control group 
members have zero preferred choices (because they are for participation in a first-choice lot- 
tery), 27.2 percent have one preferred choice, and 8.1 percent have two preferred choices. This 
equivalence holds for all 12 possible school choices. Appendix Table A.3 (see page 81) presents 
corresponding results for the 24 percent of observations that involved students who had partici- 
pated in a prior lottery. Even for this subgroup of students — which, in theory, could be affected 
by variation in their probability of prior assignment — there is no systematic difference between 
SSC lottery winners and control group members. 



10 Results for SSC lottery winners represent the simple percentage distribution of their student obser- 
vations across school choice categories. Results for control group members are computed first for each 
SSC lottery. They are then averaged across lotteries with weights proportional to each lottery’s number of 
winners. This properly accounts for lottery differences in the ratio of winners to control group members. 
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New York City Small Schools of Choice 
Appendix Table A.l 

Baseline Characteristics of SSC Lottery Participants: 
First Year of High School, Cohorts 1 to 4 



Characteristic (%) 


SSC 

Lottery 

Winners 


Control 

Group 

Members 


Estimated 

Difference 


P- Value for 
Estimated 
Difference 


Race/ethnicity 


Hispanic 


47.3 


47.9 


-0.6 


0.480 


Black 


43.6 


43.2 


0.4 


0.641 


Other 


7.9 


7.5 


0.3 


0.428 


Male 


46.0 


45.5 


0.5 


0.525 


Eligible for free/reduced-price lunch 


84.0 


84.5 


-0.5 


0.467 


English language learner 


8.4 


7.6 


0.8 


0.114 


Special education' 1 


6.6 


6.7 


-0.1 


0.826 


Overage for 8th grade b 


16.7 


18.1 


-1.4 


0.153 


8th-grade reading proficiency 0 


Did not meet standards (level 1) 


6.9 


6.6 


0.3 


0.486 


Partially met standards (level 2) 


60.5 


61.4 


-0.8 


0.328 


Fully met standards (level 3) 


28.4 


27.6 


0.8 


0.287 


Met standards with distinction (level 4) 


0.7 


0.7 


0.1 


0.580 


8th-grade math proficiency 0 


Did not meet standards (level 1) 


18.8 


19.2 


-0.3 


0.628 


Partially met standards (level 2) 


45.1 


44.9 


0.3 


0.759 


Fully met standards (level 3) 


32.8 


31.9 


0.9 


0.238 


Met standards with distinction (level 4) 


2.3 


2.2 


0.1 


0.598 


Total number of student observations = 30,959 











SOURCES: MDRC's calculations use High School Application Processing System and New York City 
Department of Education (DOE) state test data for eighth-graders from the 2004-2005 to 2007-2008 
school years, as well as data from DOE enrollment files from the 2005-2006 to 2008-2009 school 
years. 

NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for the 
difference between SSC lottery winners and control group members are obtained from a regression of a 
given baseline characteristic on a series of indicator variables that identify each lottery plus an indicator 
variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter 
indicator variable equals the difference in the mean baseline characteristic for lottery winners and 
control group members. The value for control group members equals the corresponding value for SSC 
lottery winners minus the estimated difference between lottery winners and control group members. To 
facilitate computation, all variables are centered on the mean value for the lottery they represent. This 
approach is equivalent to directly accounting for each lottery by adding a 0/1 indicator variable for it 
(Wooldridge, 2000). In some cases, rounding may cause slight discrepancies. 

(continued) 
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Appendix Table A.l (continued) 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

A chi-square test was used to assess the statistical significance of the overall difference between 
lottery winners and control group members reflected by the full set of baseline characteristics in the 
table. The resulting chi-square value is not statistically significant (p-value = 0.387). 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 
2005, 2006, 2007, and 2008, respectively. 

“This sample includes special education students who can be taught in the regular classroom setting. 
Special education students classified by the DOE as requiring collaborative team teaching services or 
self-contained classes are not part of the sample. 

b Lottery participants are classified as "overage for eighth grade" if they were 14 or older on 
September 1 of the eighth-grade school year. 

c Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade level for 
state math and reading exams. Due to missing test scores, the sum of levels 1-4 may not add to 100 
percent. 



Thus, although it is theoretically possible that differences in the relative competitiveness 
of prior school lotteries for 24 percent of sample members could produce pre-existing differences 
between SSC lottery winners and control group members for the full study sample, there is strong 
empirical evidence that this did not occur. Nevertheless, to further limit this possibility, the esti- 
mated prior probability of assignment was included as a covariate in regression-adjusted estimates 
of effects of winning an SSC lottery. This covariate had no systematic effect on the results, how- 
ever, providing further evidence that there are no pre-existing differences to control for. 1 1 

The Resulting Statistical Model 

To implement the preceding extensions of the study’s basic analytic approach, to pool 
findings across SSC lotteries, and to increase the precision of these findings, the regression 

"Empirical analysis cannot determine with certainty whether there are pre-existing differences be- 
tween SSC lottery winners and control group members because such analysis is limited to observed stu- 
dent characteristics, and it is always possible that unobserved differences exist. Nonetheless, the present 
empirical analysis is particularly telling because there is every reason to expect that if students’ counter- 
factual outcomes are prescreened by their probability of prior assignment (which is the only potential 
source of a problem), then any systematic differences in future counterfactual outcomes that result should 
be reflected as differences in eighth-grade test scores (which does not occur). 

This situation differs from nonrandomized studies that match comparison students to treatment stu- 
dents based on their pretests. In those studies, the mechanism that chooses comparison students is not the 
same as the mechanism that chooses treatment students. Thus, unobserved differences between the two 
groups can exist when they look the same on observed characteristics that are used to match them. In the 
present case, only one mechanism distinguishes between observed characteristics and unobserved charac- 
teristics of SSC lottery winners and control group members: their probability of prior assignment. If this 
mechanism prescreens SSC lottery winners and control group members differentially on their academic 
potential, it should produce systematic differences in their eighth-grade test scores. If there are no syste- 
matic differences in these test scores, there should be no systematic differences in students’ academic 
potential. 



79 




New York City Small Schools of Choice 



Appendix Table A.2 

Distribution of Lottery Participants by Their Number of Prior Choices 
Not Obtained: Cohorts 1 to 4 



Number of prior choices 
not obtained (%) 


SSC 

Lottery 

Winners 


Control 

Group 

Members 


Estimated 

Difference 


P- Value 
Estimated 
Difference 


0 


41.3 


41.3 


0.0 


0.576 


1 


27.2 


27.2 


0.0 


0.840 


2 


8.1 


8.1 


0.0 


0.973 


3 


8.0 


7.9 


0.0 


0.710 


4 


4.9 


4.9 


0.0 


0.729 


5 


3.9 


3.9 


0.0 


0.975 


6 


2.5 


2.5 


0.0 


0.818 


7 


1.5 


1.5 


0.0 


0.770 


8 


1.1 


1.1 


0.0 


0.916 


9 


0.8 


0.8 


0.0 


0.845 


10 


0.5 


0.5 


0.0 


0.901 


11 


0.2 


0.2 


0.0 


0.903 


Total number of student observations 


= 29,811 









SOURCES: MDRC's calculations use High School Application Processing System data from 
eighth-graders in 2004-2005 to 2007-2008, as well as data from New York City Department of 
Education enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for 
the difference between SSC lottery winners and control group members are obtained from a 
regression of a given baseline characteristic on a series of indicator variables that identify 
each lottery plus an indicator variable that equals one for lottery winners and zero for the lottery 
losers. The coefficient on the latter indicator variable equals the difference in the mean baseline 
characteristic for lottery winners and control group members. The value for control group 
members equals the corresponding value for SSC lottery winners minus the estimated difference 
between lottery winners and control group members. To facilitate computation, all variables are 
centered on the mean value for the lottery they represent. This approach is equivalent to directly 
accounting for each lottery by adding a zero/one indicator variable for it (Wooldridge, 2000). In 
some cases, rounding may cause slight discrepancies. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 
2005, 2006, 2007, and 2008, respectively. 
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New York City Small Schools of Choice 
Appendix Table A.3 



Distribution of Lottery Participants by Their Number of Prior Choices 
Not Obtained: Students in Cohorts 1 to 4 Who Were in Multiple Lotteries 



Number of prior choices 
not obtained (%) 


SSC 

Lottery 

Winners 


Control 

Group 

Members 


Estimated 

Difference 


P- Value 
Estimated 
Difference 


0 


0.0 


0.0 


0.0 


1.000 


1 


27.4 


27.5 


-0.1 


0.792 


2 


13.8 


13.5 


0.3 


0.292 


3 


16.4 


16.6 


-0.2 


0.546 


4 


12.4 


12.7 


-0.3 


0.220 


5 


9.9 


9.7 


0.2 


0.300 


6 


7.1 


7.4 


-0.3 


0.106 


7 


5.3 


5.0 


0.3 


0.093 


8 


3.4 


3.7 


-0.3 


0.098 


9 


2.2 


2.0 


0.2 


0.094 


10 


1.5 


1.4 


0.0 


0.733 


11 


0.7 


0.5 


0.1 * 


0.028 


Total number of student observations = 7,274 



SOURCES: MDRC's calculations use High School Application Processing System data from 
eighth-graders in 2004-2005 to 2007-2008, as well as data from New York City Department of 
Education enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for 
the difference between SSC lottery winners and control group members are obtained from a 
regression of a given baseline characteristic on a series of indicator variables that identify 
each lottery plus an indicator variable that equals one for lottery winners and zero for the lottery 
losers. The coefficient on the latter indicator variable equals the difference in the mean baseline 
characteristic for lottery winners and control group members. The value for control group 
members equals the corresponding value for SSC lottery winners minus the estimated difference 
between lottery winners and control group members. To facilitate computation, all variables are 
centered on the mean value for the lottery they represent. This approach is equivalent to directly 
accounting for each lottery by adding a zero/one indicator variable for it (Wooldridge, 2000). In 
some cases, rounding may cause slight discrepancies. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 
2005, 2006, 2007, and 2008, respectively. 
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model shown in Equation A.l was used to compare mean outcomes for SSC lottery winners 
and control group members. This type of model is often used for multisite randomized trials. 



Equation A.l: 

Yij = ^ f i + Po ' Wij + Pi ■ PPAij + Yr • Sij R (l - Mij R ) + n R • Mij r 
i 

+ Yjyj ■ S tJM (l — + TCjyi • Mljjyi + + £ij 



where: 

Yij = an outcome for participant j in lottery i 

L t = a lottery indicator equal to 1 for lottery i and 0 otherwise 

Wij = a lottery- winner indicator equal to 1 if participant j wins lottery i, and equal to 0 
otherwise 

PPA^ = the probability of prior assignment for participant j in lottery i 

S t j R and Sij M = the eighth-grade test score in reading and in math, respectively, for 
participant j in lottery i 

M t j R and M 1/M = a missing data indicator for eighth-grade test scores in reading and 
math, respectively, equal to 1 if data on the score are missing for participant j in lottery i 
and 0 otherwise 

s k = a random error for student k, assumed to be distributed independently and identi- 
cally across students 

Ei j = a random error for participant i in lottery j , assumed to be distributed indepen- 
dently and identically across observations for a student 

The first variables in Equation A. 1 are a series of 0/1 indicators, L it which identify each 
SSC lottery. These indicators account for lottery differences in the ratios of winners to control 
group members. To facilitate computation, the indicators are implemented by centering values 
of all other variables on their means for the lottery to which they apply, instead of including the 
indicators as covariates in the model. This is a common approach for estimating regression 
models with many indicator variables. 12 

The next variable in the model is a 0/1 indicator, W t j, that identifies SSC lottery win- 
ners. The estimated coefficient, /? 0 , for this variable is the regression-adjusted difference be- 
tween mean outcomes for lottery winners and control group members. This result is the esti- 
mated effect of winning an SSC lottery. The standard error and statistical significance level (p- 



12 See Wooldridge (2000). 
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value) for this coefficient are the standard error and statistical significance level for the esti- 
mated effect. For reference, Appendix Tables B.l through B.4 present estimated effects of win- 
ning an SSC lottery for each outcome and sample in the present report. 

The next variable in the model is the probability of prior assignment, PPAij. Its value is 
equal to 0 for the 76 percent of student observations that are not in a prior lottery, and its value 
ranges from just above 0 to just below 1 for the 24 percent of student observations that are in a 
prior lottery. 

Lastly, student scores on eighth-grade reading and math tests (S L j R and S ijM ) are in- 
cluded to increase the precision of the analysis. Zero/one indicators (M ijR and M ijM ) are in- 
cluded for these test scores to account for their missing data for some students. 13 



Converting Estimated Effects of Winning an SSC Lottery into 
Estimated Effects of Enrolling in an SSC 

Chapter 2 indicates that for many students who participate in multiple SSC lotteries, 
losing a lottery is not the same as not being assigned by HSAPS to an SSC. This is because 
these students can win another lottery for a different SSC and be assigned to it by HSAPS. They 
also can be assigned by HSAPS to another SSC without participating in another lottery. This 
can occur either because the other SSC is not oversubscribed (so that all of its potential as- 
signees are accepted) or because it is oversubscribed but has sufficient capacity to accept all 
potential assignees in the students’ priority cell. 

Because of this, the difference between winning and losing an SSC lottery is not the 
same as the difference between being assigned to an SSC and not being assigned to an SSC. For 
these students, then, the average effect of winning an SSC lottery (as opposed to losing the lot- 
tery) is not the same as the average effect of being assigned to an SSC (as opposed to not being 
assigned to an SSC). This means that the average effect of winning an SSC lottery is not the 
same as the average effect of “intent to treat,” which is what most randomized studies report. 
Indeed, it is not clear what the average effect of winning an SSC lottery means. 

This ambiguity makes it necessary to convert estimates of average effects of winning an 
SSC lottery into estimates of something that is more meaningful: average effects of enrolling in 
an SSC. This latter type of effect is an example of what is referred to in the statistics literature as 
a local average treatment effect. 14 For the present analysis, this local average treatment effect 

13 In the four-cohort sample for the ninth-grade on-track indicator discussed in Chapter 3, 3.4 percent 
of student observations are missing data for their eighth-grade reading test, and 0.8 percent are missing 
data for their eighth-grade math test. There is no statistically significant difference in these missing data 
rates for SSC lottery winners and control group members. 
l4 See Angrist, Imbens, and Rubin (1996). 
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was estimated using what is now a standard application of instrumental variables analysis to a 
randomized trial. 15 

The following sections describe the instrumental variables model used, present the 
conceptual rationale for this model, examine its assumptions, and explain how its findings are 
reported. 

Statistical Model 

The present instrumental variables model produces estimates of average effects of enrol- 
ling in an SSC for students who enroll in an SSC because they win a lottery for that SSC. The 
members of this type of subgroup are typically referred to as “compilers” in the statistics litera- 
ture. 16 The present report does not use this label, however, because it is not clear what “com- 
pliance” means for students who lose one SSC lottery and win another. These students can be 
noncompliers for their first lottery (because they eventually enroll in an SSC even though they 
lose a prior SSC lottery) and compliers for their second lottery (because they win their second 
lottery and enroll in that SSC). The report refers to this subgroup as “target SSC enrollees” be- 
cause (as explained below) its members are the target of estimates of SSC enrollment effects. 

Estimates of SSC enrollment effects for target SSC enrollees were obtained from two- 
stage least squares estimation of an instrumental variables model that includes a separate in- 
strument for each lottery. In effect, this model uses two-stage least squares to estimate an SSC 
enrollment effect for each lottery, and pools resulting estimates across lotteries. The instru- 
ment for each lottery is an interaction between the 0/1 indicator (Wy) that distinguishes SSC 
lottery winners from control group members and the 0/1 indicator (L,) that identifies the lot- 
tery involved. 

This approach has been used for past analyses of multisite randomized experiments and 
is becoming popular for use with randomized studies in general and lottery-based studies in par- 
ticular. 17 The instruments used for the present analysis are valid because they are randomized 
and thus cannot be correlated with error terms for student outcomes. The instruments are 
“strong” because winning or losing an SSC lottery is highly correlated with enrolling or not 
enrolling in an SSC. 18 The resulting two-stage least squares model is as follows. 



15 For instrumental variables analysis, see Angrist, Imbens, and Rubin (1996). 

16 See Angrist, Imbens, and Rubin (1996). 

n For use of this approach with multisite randomized experiments, see Gennetian, Morris, Bos, and 
Bloom (2005) and Ludwig and Kling (2007); with randomized studies in general, see Angrist and Pischke 
(2009); and with lottery-based studies in particular, see Abulkadiroglu et al. (2009). 

18 See Bloom, Zhu, and Unlu (2010) or Angrist and Pischke (2009) for a discussion of instrument 
strength and finite sample bias. To avoid this bias, present results are only reported for models with a 
first-stage F statistic equal to 10 or more, as recommended by Stock and Yogo (2005). 
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First-Stage Equation 

The first-stage equation in the model specifies enrollment or not in an SSC (by the fol- 
low-up year represented by the outcome measure) as a function of indicator variables that iden- 
tify each lottery, an indicator variable that distinguishes between lottery winners and control 
group members, students’ prior probabilities of assignment, students’ pretest scores on eighth- 
grade standardized tests of reading and math, plus a missing data indicator for each pretest 
score. The first-stage equation is thus: 

Equation A.2: 

z ij = a i • Li + ^ /?o i • w ij • Li +/?1 • PPAij + y R ■ S iJR ( 1 - M iJR ) + n R ■ Mij R 

i i 

T Ym ' S ijM (l — jM ) + ' SijM T v k + v tJ 

where: 

Z ( y = an SSC enrollment indicator equal to 1 if participant j in lottery i enrolled in an 
SSC by the follow-up year represented by the outcome measure and 0 otherwise 

L l = a lottery indicator equal to 1 for lottery i and 0 otherwise 

W t j = a lottery-winner indicator equal to 1 if participant j won lottery i and 0 otherwise 

PPA^ = the probability of prior assignment for participant j in lottery i 

Sij R and S L j M = scores on eighth-grade tests of reading and math, respectively, for par- 
ticipant j in lottery i 

M L j R and M ijM = missing data indicators for eighth-grade tests of reading and math, re- 
spectively, equal to 1 if data are missing for participant j in lottery i and 0 otherwise 

v k = a random error for student k, assumed to be distributed independently and identical- 
ly across students 

Vij = a random error for participant i in lottery j, assumed to be distributed indepen- 
dently and identically across observations for a student 

For this analysis, SSC enrollment is defined as whether or not students had enrolled in 
an SSC at any time during or before the follow-up year represented by the outcome measure. 
Thus, for estimates of SSC effects on ninth-grade outcomes, SSC enrollment is defined as 
whether or not students had enrolled in an SSC at any time during their first year of high school. 
For estimates of SSC effects on four-year graduation rates, SSC enrollment is defined as wheth- 
er or not students had enrolled in an SSC at any time during their first four years of high school. 
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Second Stage Equation 

The second-stage equation in the model specifies the outcome, Y^, for each lottery par- 
ticipant as a function of the predicted value of his SSC enrollment indicator, Z ^ , which equals 
the fitted value of the indicator from the first-stage equation. In addition, it is necessary to in- 
clude in the second- stage equation all exogenous independent variables from the first stage. 19 
The second-stage equation is thus: 

Equation A.3: 

Yij = Ei 0i ' + 9 0 ■ Zij + • PPAij + p R ■ S ijR (l - M ijR ) + u) R ■ M R 

+ Pm ' s m(J- ~ M ijM ) + ' M ijM + £ ic + £ 'ij 

where: 

s' k = a random error for student k, assumed to be distributed independently and 

identically across students 

s'ij = a random error for participant i in lottery j, assumed to be distributed indepen- 
dently and identically across observations for a student 

The estimated value of the enrollment coefficient (0 O ) in the second-stage equation is 
the estimated average effect of enrolling in an SSC for target SSC enrollees (their local average 
treatment effect). The estimated standard error and p-value for this coefficient are the estimated 
standard error and p-value for the estimated effect. 

Conceptual Rationale 

Appendix Figure A.3 illustrates sample subgroups that are the conceptual basis for 
converting estimated effects of winning an SSC lottery into estimated effects of SSC enroll- 
ment through instrumental variables analysis. This figure represents a single hypothetical SSC 
lottery, with a bar on the left side for lottery winners and a bar on the right side for control- 
group members. 

The top segment of the bar for lottery winners represents members of this group who do 
not enroll in an SSC and thereby become “no-shows.” 20 Only a small fraction of lottery winners 
are no-shows, as suggested by the small section of the bar that is allotted to this group. The top 
segment of the bar depicting the control group represents “no-show counterparts.” These are 
students who lose the SSC lottery, do not enroll in an SSC, and would not have enrolled in one 
if they had won the lottery. 



l9 See Angrist and Pischke (2009). 
20 See Bloom (1984). 
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New York City Small Schools of Choice 
Appendix Figure A.3 

A Model of SSC Enrollment Among Hypothetical SSC Lottery 
Winners and Control Group Members 

SSC Lottery Control Group 

Winners Members 



No-shows 

(Do not enroll in SSC) 




No-show counterparts 

(Do not enroll in SSC) 


Target SSC enrollees 

(Enroll in SSC) 




Control group counterparts 
for target SSC enrollees 

(Do not enroll in SSC) 


Non-HSAPS crossover counterparts 

(Do not enroll in SSC) 




Non-HSAPS crossovers 

(Enroll in SSC) 


HSAPS crossover counterparts 

(Do not enroll in SSC) 




HSAPS crossovers 

(Enroll in SSC) 



Randomization ensures that the proportion of control group members who are no-show 
counterparts equals (in expectation) the proportion of SSC lottery winners who are no-shows. 
Randomization also ensures that mean counterfactual outcomes for no-show counterparts equal 
(in expectation) mean counterfactual outcomes for no-shows. It is possible to identify individual 
no-shows but it is not possible to identify individual no-show counterparts. However, all that is 
necessary for the analysis (as will be seen) is the statistical fact that no-shows and no-show 
counterparts have the same mean counterfactual outcomes and make up the same proportion of 
their experimental groups. 

The bottom two segments of the control group bar represent control group members 
who, although they lose the present lottery, ultimately enroll in an SSC and thereby become 
“crossovers .” 21 It is possible to identify individual crossovers among control group members but 
it is not possible to identify individual crossover counterparts among SSC lottery winners. Once 
again, all that is necessary for the analysis is that the relative sizes and mean counterfactual out- 



21 Bloom et al. (1996). 
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comes are the same for the corresponding subgroups. The bottom-most segment of the control 
group bar represents students who lose the present SSC lottery, are assigned by HSAPS to 
another SSC, and ultimately enroll in an SSC. This subgroup is referred to as “HSAPS crossov- 
ers” because their crossover status is caused by HSAPS. Randomization ensures that among 
winners of the present SSC lottery there is a subgroup of “HSAPS crossover counterparts” 
whose relative size and mean counterfactual outcomes are the same (in expectation) as those for 
HSAPS crossovers. 

The second-to-bottom segment of the control group bar in Figure A.3 represents stu- 
dents who lose the current SSC lottery but enroll in an SSC through channels outside of 
HSAPS. (For example, a student may move into the district or transfer from a parochial school 
after the HSAPS process has concluded. In those instances, a student enrolls through a borough 
enrollment office.) This subgroup is referred to as “non-HSAPS crossovers.” Randomization 
ensures that among winners of the present SSC lottery there is a subgroup of “non-HSAPS 
crossover counterparts” whose relative size and mean counterfactual outcomes are the same (in 
expectation) as those for non-HSAPS crossovers. Note that there are considerably more HSAPS 
crossovers than non-HSAPS crossovers, which is the case for the study sample. 

The remaining (and by far the largest) subgroups of lottery winners and control group 
members who are depicted in Figure A.3 are target SSC enrollees and target SSC enrollee coun- 
terparts. As noted, these sample members are typically referred to as compliers. But given the 
ambiguity of the concept of compliance in the present situation, this label is not used. 

Randomization ensures that the relative sizes and mean counterfactual outcomes of tar- 
get SSC enrollees and their control group counterparts are the same in expectation. And as 
shown below, these are the only sample members whose high school experiences provide in- 
formation about the effects of SSC enrollment. Hence, they are the only students for whom SSC 
enrollment effects can be estimated. 

To do so requires the following assumptions: 

• Assignment to an SSC by itself (without actually enrolling in the SSC) does 
not affect student outcomes. 22 

• Average effects of SSC enrollment are approximately the same (zero) for no- 
shows and no-show counterparts (because neither enrolls in an SSC). 

• Average effects of SSC enrollment are approximately the same for crossov- 
ers and crossover counterparts (because both enroll in an SSC). 



2 This assumption is often called the “exclusion restriction” ( Angrist, Imbens, and Rubin, 1996). 




The preceding assumptions plus the counterfactual equivalence of the corresponding 
subgroups in Figure A.3, which is ensured by randomization, imply that in expectation there is 
zero difference in mean future outcomes for no-shows and no-show counterparts and there is 
zero difference in mean future outcomes for crossovers and crossover counterparts. Therefore, 
in expectation these sample members contribute zero difference to any observed difference in 
mean outcomes for the full sample of SSC lottery winners and control group members. The only 
potential source of any difference is target SSC enrollees (who enroll in an SSC) and their con- 
trol group counterparts (who do not enroll in an SSC). 

Consequently, the expected value of the observed difference in mean outcomes for SSC 
lottery winners and control group members is a weighted average of zero difference for no- 
shows and their counterparts plus zero difference for crossovers and their counterparts plus the 
average effect of SSC enrollment for target SSC enrollees and their counterparts. Weights for 
this average are proportional to the size of each subgroup. In symbols: 

Equation A.4: 

Ew = E ns • 0 + P HCC • 0 + P NHC c • 0 + (1 — P NS — P HC c ~ Pnhcc ) ' E e 
= (1 — Pns ~ Phcc ~ Pnhcc ) ’ Ee 

where: 



E w and E e = the expected value of the average effect of winning an SSC lottery for all 
lottery participants and the expected value of the average effect of enrolling in an SSC 
for target SSC enrollees, respectively 

P NS , P H cc> and Pnhcc = the expected proportions of SSC lottery winners who are no- 
shows, HSAPS crossover counterparts, and non-HSAPS crossover counterparts, respec- 
tively 

Therefore, 



Equation A.5: 

E e ~ 



‘W 



(1 - P 



NS 



HCC 



NHCCJ 



Equation A.5 illustrates the simplest possible form of instrumental variables analysis, a 
Wald estimator. 23 For the present analysis, this estimator equals the ratio of the effect of winning 
an SSC lottery on students’ outcomes (E w ) to the proportion of SSC lottery winners who are 
target SSC enrollees (1 — P NS — P HC c ~ Pnhcc)- 24 For example, if the effect of winning the 



23 See Wald (1940). 

24 The proportion of SSC lottery winners who are target SSC enrollees is equivalent to the effect of 
winning an SSC lottery on the probability of enrolling in an SSC. 



89 




lottery were an increase of 7 percentage points in the likelihood of making adequate progress 
toward graduation, and the proportion of lottery winners who are target SSC enrollees were 0.5, 
then the effect of enrolling in an SSC would be (7/0.5), or 14 percentage points. This result 
could also be obtained using two-stage least squares analysis with a single instrument (winning 
the lottery or not winning the lottery). 



Bridging the Gap Between the Statistical Model and the Implications of Its 
Conceptual Rationale 

The preceding conceptual rationale and its corresponding Wald estimator are for a sin- 
gle SSC lottery with a single instrumental variable, whereas the present statistical model is for 
297 lotteries with 297 instrumental variables. Thus, a simple Wald estimator is not used for the 
present analysis although its basic intuition holds for each SSC lottery included. 

Consequently, for this analysis the relationship between estimated effects of winning an 
SSC lottery (E w ) and estimated effects of enrolling in an SSC (E e ) is not the same as that for a 
single lottery. To see this point, first rearrange Equation A.5 and substitute into it estimates of 
the two effects represented by E w and E e . This yields the following equation. 



Equation A.6: 



1 — Pns ~ Phcc ~~ Pnhcc 



Ew 

Ee 



Equation A.6 indicates that for a single lottery, the proportion of SSC lottery winners 
who are target SSC enrollees can be estimated from the ratio of the estimated effect of winning 
an SSC lottery (obtained using ordinary least squares) and the estimated effect of enrolling in an 
SSC (obtained using two-stage least squares with a single instrument). 



However, as noted, this simple relationship does not exist for analyses of multiple lot- 
teries with multiple instruments. 26 In fact, this relationship is quite complex and could be a 
source of confusion if direct comparisons are made between estimates of SSC enrollment ef- 
fects on student outcomes reported in Chapter 3 and corresponding estimates of effects of win- 
ning an SSC lottery presented in Appendix B. 



For example, the ratio of these estimates for the ninth-grade on-track indicator equals 
0.74. This implies that for these outcomes the effects of enrolling in an SSC are (1/0.74) or 1.35 



26 This issue arises because SSC enrollment effects (which are local average treatment effects) vary 
across lotteries, and the two-stage least squares approach, with a separate instrument for each lottery, 
produces a complex weighted average of estimates of these effects. Although this issue goes beyond the 
scope of the present study, the fact that such variation exists and is statistically significant was established 
using a “random effects” model. 
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times the size of effects of winning an SSC lottery. This can be confirmed by comparing esti- 
mates in Table 3.2 of Chapter 3 with their counterparts in Table B. 1 of Appendix B. 

Now recall that approximately 58 percent of all lottery winners in the ninth-grade on- 
track sample are target SSC enrollees. If the study sample were considered as a single com- 
posite lottery with a single instrument (which is possible to do but was judged not to be as ap- 
propriate as specifying a separate instrument for each lottery), the estimated effect of enrolling 
in an SSC would be (1/0.58) or 1.72 times the estimated effect of winning an SSC lottery. This 
larger multiple would produce estimates of SSC enrollment effects that are larger than those 
presented in this report. Indeed, one of the reasons that multiple instruments were used instead 
of a single instrument was a desire to be conservative about the size of the ratio that was used to 
convert estimates of winning an SSC lottery into estimates of enrolling in an SSC. 

Plausibility of the Assumptions Used 

To assess the validity of findings in this report, it is important to consider the plausibili- 
ty of the assumptions upon which they are based. Recall that the first such assumption is that, by 
itself, HSAPS assignment of a student to a particular school (without the student actually enrol- 
ling in that school) has no effect on that student’s future academic success. This assumption 
seems almost self-evident for no-shows who are assigned by HSAPS to an SSC but do not 
enroll in one. And it should apply with equal force to no-show counterparts. However, as dis- 
cussed below, its plausibility is less obvious for crossovers. 

The second assumption noted above is that no-shows and no-show counterparts expe- 
rience the same average effect of enrolling in an SSC — zero effect. This assumption also 
seems self-evident because no members of either group enroll in an SSC and hence they do not 
experience an SSC as students. 

The third assumption is that SSC enrollment effects are the same for crossovers in the 
control group and their counterparts among SSC lottery winners. The plausibility of this as- 
sumption should be considered separately for HSAPS crossovers and non-HSAPS crossovers. 

Recall that non-HSAPS crossovers lose an SSC lottery and are not assigned by HSAPS 
to an SSC, but somehow manage to enroll in one. In contrast, their counterparts among SSC 
lottery winners are assigned by HSAPS to an SSC and simply enroll in the SSC to which they 
are assigned. To the extent that non-HSAPS crossovers must expend extra effort to enroll in an 
SSC, they may be especially motivated by getting what they worked for. In addition, it is possi- 
ble that the school they wanted to attend is an especially good fit for them. 

If either or both of these conditions exist, then non-HSAPS crossovers may experience 
larger SSC enrollment effects than do their counterparts among SSC lottery winners. If this is 
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the case, non-HSAPS crossovers and their counterparts will cause current estimates to under- 
state the true effects of enrolling in an SSC. 27 

Although there is no direct empirical evidence about this issue, there is one piece of 
indirect evidence: the fact that estimated effects for students who are known to an SSC are no 
larger than those for students who are not known. Since some effort is involved in making 
oneself known to an SSC, this finding seems relevant for non-HSAPS crossovers and sug- 
gests that the third assumption above — that SSC enrollment effects are the same for non- 
HSAPS crossovers in the control group and their counterparts among SSC lottery winners — 
might hold for them. 

Now recall that HSAPS crossovers lose an SSC lottery and are assigned by HSAPS to 
another SSC that is farther down in their list of choices. If school choice per se makes a positive 
difference in students’ academic success, those students who enroll in a higher-choice SSC will 
experience larger enrollment effects. (To some extent this might be considered a violation of the 
first assumption described above that, by itself, HSAPS assignment of a student to a particular 
school has no effect on that student’s future academic success). This violation of the third as- 
sumption would cause current estimates to overstate the true effects of SSC enrollment. 28 

Chapter 3 presents empirical evidence that suggests that it is unclear whether the effects 
of SSC enrollment vary systematically across the rank-ordered preference of the SSC to which 
students are assigned. Overall, there is not statistically significant variation in SSC effects 
among students who ranked the SSC to which they were assigned as their first choice, their 
second or third choice, or their fourth through twelfth choice. It does appear, however, that the 
effect of SSC enrollment may be smaller for students who rank the SSC to which they were 
assigned as their first-choice school relative to the SSC effects for all other students. If that is 
the case, then present estimates would understate the true effects of SSC enrollment on target 
SSC enrollees. 

On balance, then, existing empirical evidence suggests that the present estimates do not 
overstate the effects of SSC enrollment. 



27 Instead of the assumed zero difference between mean outcomes for non-HSAPS crossovers and 
their counterparts among lottery winners, there would be a negative difference. This negative difference 
would be embedded in the observed difference in mean outcomes for all lottery winners and control 
group members, which would cause the observed difference to understate the true difference produced by 
target SSC enrollees and their control group counterparts. 

28 The situation here is the reverse of that in note 27. 
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Reporting the Findings 

Table A.4 is a portion of Table 3.2 from Chapter 3. This table illustrates how estimates 
of SSC enrollment effects are reported. The first column in the table shell is for estimates of 
mean outcomes for target SSC enrollees; the second column is for estimates of mean outcomes 
for control group counterparts of target SSC enrollees; the third column is for estimates of dif- 
ferences between mean outcomes for the two groups; and the fourth column is for p-values of 
these estimated differences. 

Findings for the last two columns are obtained directly from estimates of 9 0 in Equation 
A.3 and its p-value. For example, the last two numbers in the first row of the table indicate that 
the estimated effect of SSC enrollment on the likelihood of target SSC enrollees being on track 
toward graduation at the end of ninth grade is an increase of 10.0 percentage points with a p- 
value of 0.000 (which is highly statistically significant). Findings for the first column, which 
represent the mean value of each outcome measure for target SSC enrollees, are estimated as 
described below. The first number in the first row of the table indicates that 58.5 percent of tar- 
get SSC enrollees were estimated to be on track toward graduation at the end of ninth grade. 
Findings for the second column, which represent the mean value of each outcome measure for 
the control group counterparts of target SSC enrollees, are obtained by subtracting the corres- 
ponding value in the third column (“Estimated effect”) from that in the first column. For exam- 
ple, the second number in the first row of the table (48.5) was obtained by subtracting the third 
number (10.0) from the first number (58.5). This result indicates that 48.5 percent of the control 
group counterparts of target SSC enrollees were estimated to be on track toward graduation at 
the end of ninth grade. 

The present approach to reporting results has been used in previous MDRC 
studies. 29 What differs from previous studies is that mean outcomes for this study must be 
reported for a subgroup of treatment group members (target SSC enrollees) who cannot be 
identified individually. 30 Fortunately, the statistical properties of randomization make it 
possible to obtain valid estimates of mean outcomes for this subgroup. All that is needed is 
information about the proportion of SSC lottery winners who enroll in an SSC ( Plwe = 1 — 
Pns)> the observed mean outcome for these enrollees (Y LWE ), the proportion of control 



29 See Black et al. (2009); Corrin et al. (2008). 

30 Because most randomized studies only report estimates of average effects of intent to treat, they 
can present observed mean outcomes for all treatment group members as a contextual anchor for their 
estimates of treatment effects. However, because this type of effect does not have a meaningful interpre- 
tation for the present study, it was necessary instead (as noted above) to report estimates of local average 
treatment effects for target SSC enrollees. Since it is not possible to identify individual compilers, it is not 
possible to measure their mean outcomes directly. Instead, they must be estimated from observed out- 
comes for all treatment group members who receive treatment and all crossovers (as described in this 
section). 
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New York City Small Schools of Choice 



Appendix Table A.4 

Sample Partial Table of Estimated SSC Enrollment Effects 



Outcome 


Target 

SSC 

Enrollees 


Control 

Group 

Counterparts 


Estimated 

Effect 


P-V alue for 
Estimated 
Effect 


9th-grade on-track indicator 3 (%) 


58.5 


48.5 


10.0 ** 


0.000 


Earned 10 or more credits 


73.1 


62.3 


10.8 ** 


0.000 


Failed more than 1 semester of a core subject 


39.0 


46.8 


-7.8 ** 


0.000 


Total number of student observations = 29,81 1 



SOURCES: MDRC's calculations use High School Application Processing System data from eighth- 
graders in 2004-2005 to 2007-2008, as well as data from New York City Department of Education 
attendance, course credits, and enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: This table presents the estimated effects for students who have follow-up course credits data. 
Appendix A describes how values in the column labeled "Target SSC Enrollees" are estimated. 
Appendix A also describes how values in the column labeled "Estimated Effect" are estimated. 

Values in the column labeled "Control Group Counterparts" are differences between corresponding 
values in the first and third columns. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated 
as: ** = 1 percent; * = 5 percent. 

a The on-track composite measure indicates whether students earned at least 10 credits in their first 
year of high school and had no more than one semester of failure in a core subject in that school year 
(English, math, science, and social studies). 



group members who are crossovers (P co ), and the observed mean outcome for those cross- 
overs (? co ). Randomization ensures that P(:o and Y co are unbiased estimates of corresponding 
parameters for crossover counterparts among SSC lottery winners. 

The next step is to note that a mean outcome for all SSC lottery winners who enroll in 
an SSC ( Y lwe ) is a weighted average of the corresponding mean outcome for target SSC enrol- 
lees (Y tarE ) and crossover counterparts with weights equal to the size of each group. Then, note 
that the mean outcome for crossovers ( Yq ) is an unbiased estimate of the mean outcome for 
crossover counterparts. Together, these facts imply that: 



Equation A.7: 



Ylwe — 



1 -Pns~ Pi 



CO 



1 -p, 



NS 



'tarE 



+ 



CO 



Ll -P, 



NS J 



l co 



Solving Equation A.7 for Y tarE yields: 
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Equation A.8: 



^ tarE 



1 -P, 



NS 



1 -P NS ~P. 



CO 



1 LWE 



+ 



CO 



1-Pns-P. 



CO 



'CO 



In this way, the implied value of Y tarE can be estimated from observable values of 
P NS , P co , Y lwe , and Y co . This approach also works for estimating mean values of background 
characteristics of target SSC enrollees (X tarE ). 
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Appendix B 

Effects of Winning an SSC Lottery 




New York City Small Schools of Choice 
Appendix Table B.l 



Estimated Effects of Winning an SSC Lottery on the Transition into High School: 
First Year of High School, Cohorts 1 to 4 



Outcome 


SSC Control 
Lottery Group Estimated 

Winners Members Effect 


Effect Size 
(Standard 
Deviation) 


P -Value for 
Estimated 
Effect 


Course credits 












Ninth-grade on-track indicator 3 (%) 


57.9 


50.5 


7.4 


** 


0.000 


Earned 1 0 or more credits (%) 


72.0 


64.5 


7.4 


** 


0.000 


Failed more than 1 semester of a core subject (%) 39.8 


46.1 


-6.3 


** 


0.000 


Total credits earned toward graduation 13 


11.2 


10.5 


0.7 


** 0.17 


** 0.000 


Attendance t%l 












Overall attendance rate 


87.1 


86.2 


1.0 


** 


0.000 


Regular attendance rate (90 percent or higher) 


60.2 


56.0 


4.2 


** 


0.000 


Total number of student observations = 29,8 1 1 



SOURCES: MDRC's calculations use High School Application Processing System (HSAPS) data from 
eighth-graders in 2004-2005 to 2007-2008, as well as data from New York City Department of Education 
attendance, course credits, and enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: This table presents the estimated effects for students who have follow-up course credits data. 
Estimated differences between SSC lottery winners and control group members are regression-adjusted as 
described by Equation A. 1 in Appendix A. Variables in the regression are measured as deviations from their 
lottery mean in order to account for the lottery for each sample point. This approach is equivalent to directly 
accounting for each lottery by adding a zero/one indicator variable for it (Wooldridge, 2000). Values in the 
column labeled "SSC Lottery Winners" are observed means for lottery winners assigned by HSAPS to an 
SSC. Values in the column labeled "Control Group Members" are regression-adjusted means that match the 
distribution of SSC lottery winners across lotteries. In some cases, rounding may cause slight discrepancies 
when comparing the statistical significance of findings in this table (for effects of winning an SSC lottery) 
with the statistical significance of findings in Table 3.2 (for effects of enrolling in an SSC). 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated as: ** = 

1 percent; * = 5 percent. 

The estimated effect size is calculated as a proportion of the standard deviation of the outcome for control 
group members. 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 2005, 2006, 
2007, and 2008, respectively. 

a The on-track composite measure indicates whether students earned at least 10 credits in their first year of 
high school and had no more than one semester of failure in a core subject in that school year (English, math, 
science, and social studies). 

b The "total credits earned toward graduation" measure is the aggregate number of course credits earned 
toward fulfilling the New York State graduation requirements. The credit requirements are as follows: 3 1 
core subject credits, including 8 credits each of English and social studies; 6 credits each of math and science; 

2 credits of arts; 1 credit of health; and 13 additional credits, including 4 credits of physical education, 2 
credits of a foreign language, and 7 credits of electives. 



99 








New York City Small Schools of Choice 
Appendix Table B.2 

Estimated Effects of Winning an SSC Lottery on Progress Toward Graduation: 
Second Year of High School, Cohorts 1 to 3 

SSC Control Effect Size P-Value for 

Lotteiy Group Estimated (Standard Estimated 
Outcome Winners Members Effect Deviation) Effect 

Course credits 



Earned 20 or more credits (%) 


68.8 


62.8 


6.1 ** 


0.000 


Total credits earned toward graduation 2 


22.2 


20.8 


14 ** 


0.16 ** 0.000 


Resents exams 










Total Regents exams passed toward graduation 11 


1.5 


1.4 


0.0 


0.03 0.070 


Attendance (%) 










Overall attendance rate 


84.7 


83.4 


1.3 ** 


0.000 


Regular attendance rate (90 percent or higher) 


55.2 


52.1 


3.1 ** 


0.001 



Total number of student observations = 21,822 

SOURCES: MDRC's calculations use High School Application Processing System (E1SAPS) data from 
eighth-graders in 2004-2005 to 2006-2007, as well as data from New York City Department of Education 
attendance, course credits, Regents exam, and enrollment files from the 2005-2006 to 2008-2009 school 
years. 

NOTES: This table presents the estimated effects for students who have follow-up course credits data. 
Estimated differences between SSC lottery winners and control group members are regression-adjusted as 
described by Equation A. 1 in Appendix A. Variables in the regression are measured as deviations from their 
lottery mean in order to account for the lottery for each sample point. This approach is equivalent to directly 
accounting for each lotteiy by adding a zero/one indicator variable for it (Wooldridge, 2000). Values in the 
column labeled "SSC Lottery Winners" are observed means for lottery winners assigned by E1SAPS to an 
SSC. Values in the column labeled "Control Group Members" are regression-adjusted means that match the 
distribution of SSC lottery winners across lotteries. In some cases, rounding may cause slight discrepancies 
when comparing the statistical significance of findings in this table (for effects of winning an SSC lotteiy) 
with the statistical significance of findings in Table 3.4 (for effects of enrolling in an SSC). 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated as: ** = 

1 percent; * = 5 percent. 

The estimated effect size for each measure is calculated as a proportion of the standard deviation of the 
outcome for control group members. 

Cohorts 1, 2, and 3 consist of students in the study who were eighth-graders in the spring of 2005, 2006, 
and 2007, respectively. 

a The "total credits earned toward graduation" measure is the aggregate number of course credits earned 
toward fulfilling the New York State graduation requirements. The credit requirements are as follows: 31 
core subject credits, including 8 credits each of English and social studies; 6 credits each of math and science; 

2 credits of arts; 1 credit of health; and 13 additional credits, including 4 credits of physical education, 2 
credits of a foreign language, and 7 credits of electives. 

b In order to receive a New York State diploma, students must pass the following core subject Regents 
exams: English Language Arts, Math A, U.S. Elistory, Global Elistory, and one of the science exams 
(Chemistry, Physics, Earth Science, Biology, or Living Environment). The "total Regents exams passed 
toward graduation" measure counts the number of required Regents exams that a student has passed with a 
score of 65 or above. 
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New York City Small Schools of Choice 
Appendix Table B.3 

Estimated Effects of Winning an SSC Lottery on Progress Toward Graduation: 
Third Year of High School, Cohorts 1 and 2 



Outcome 


SSC 

Lottery 

Winners 


Control 

Group 

Members 


Effect Size 
Estimated (Standard 
Effect Deviation) 


P-Value for 
Estimated 
Effect 


Course credits 












Earned 30 or more credits (%) 


69.5 


65.5 


3.9 ** 




0.000 


Total credits earned toward graduation 11 


32.1 


30.8 


1.3 ** 


0.12 


** 0.000 


Regents exams 












Total Regents exams passed toward graduation 11 


2.7 


2.6 


0.1 ** 


0.06 


** 0.003 


Attendance (%) 












Overall attendance rate 


82.3 


80.7 


1.7 ** 




0.001 


Regular attendance rate (90 percent or higher) 


51.4 


47.4 


4.0 ** 




0.001 



Total number of student observations = 13,297 



SOURCES: MDRC's calculations use High School Application Processing System (HSAPS) data from eighth- 
graders in 2004-2005 and 2005-2006, as well as data from New York City Department of Education 
attendance, course credits, Regents exam, and enrollment files from the 2005-2006 to 2008-2009 school years. 

NOTES: This table presents the estimated effects for students who have follow-up course credits data. 
Estimated differences between SSC lottery winners and control group members are regression-adjusted as 
described by Equation A. 1 in Appendix A. Variables in the regression are measured as deviations from their 
lottery mean in order to account for the lottery for each sample point. This approach is equivalent to directly 
accounting for each lottery by adding a zero/one indicator variable for it (Wooldridge, 2000). Values in the 
column labeled "SSC Lottery Winners" are observed means for lottery winners assigned by HSAPS to an SSC. 
Values in the column labeled "Control Group Members" are regression-adjusted means that match the 
distribution of SSC lottery winners across lotteries. In some cases, rounding may cause slight discrepancies 
when comparing the statistical significance of findings in this table (for effects of winning an SSC lottery) with 
the statistical significance of findings in Table 3.5 (for effects of enrolling in an SSC). 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated as: ** = 1 
percent; * = 5 percent. 

The estimated effect size for each measure is calculated as a proportion of the standard deviation of the 
outcome for control group members. 

Cohorts 1 and 2 consist of students in the study who were eighth-graders in the spring of 2005 and 2006, 
respectively. 

a The "total credits earned toward graduation" measure is the aggregate number of course credits earned 
toward fulfilling the New York State graduation requirements. The credit requirements are as follows: 31 core 
subject credits, including 8 credits each of English and social studies; 6 credits each of math and science; 2 
credits of arts; 1 credit of health; and 13 additional credits, including 4 credits of physical education, 2 credits 
of a foreign language, and 7 credits of electives. 

b ln order to receive a New York State diploma, students must pass the following core subject Regents 
exams: English Language Arts, Math A, U.S. History, Global History, and one of the science exams 
(Chemistry, Physics, Earth Science, Biology, or Living Environment). The "total Regents exams passed toward 
graduation" measure counts the number of required Regents exams that a student has passed with a score of 65 
or above. 
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New York City Small Schools of Choice 
Appendix Table B.4 



Estimated Effects of Winning an SSC Lottery on Graduation: 
Fourth Year of High School, Cohort 1 



Outcome (%) 


SSC 
Lottery 
W inners 


Control 

Group Estimated 
Members Effect 


P-Value for 
Estimated 
Effect 


Graduation 










Graduated from high school 


68.1 


63.8 


4.4 ** 


0.006 


Local diploma granted 


24.2 


21.8 


2.5 


0.082 


Regents diploma granted 


38.9 


36.8 


2.2 


0.170 


Advanced Regents diploma granted 


4.7 


5.2 


-0.5 


0.425 


College readiness 










Math A Regents exam score of 75 or above 


22.2 


22.2 


0.0 


1.000 


English Regents exam score of 75 or above 


33.6 


30.6 


3.0 * 


0.022 


Attendance 










Overall attendance rate 


80.8 


79.6 


1.2 


0.121 


Regular attendance rate (90 percent or higher) 


42.9 


42.1 


0.7 


0.669 


Total number of student observations = 5,363 



SOURCES: MDRC's calculations use High School Application Processing System (HSAPS) data 
from eighth-graders in 2004-2005, as well as data from New York City Department of Education 
attendance, course credits, Regents exam, transactional, and enrollment files from the 2005-2006 to 
2008-2009 school years. 

NOTES: This table presents the estimated effects for students who have follow-up course credits 
data. Estimated differences between SSC lottery winners and control group members are regression- 
adjusted as described by Equation A. 1 in Appendix A. Variables in the regression are measured as 
deviations from their lottery mean in order to account for the lottery for each sample point. This 
approach is equivalent to directly accounting for each lottery by adding a zero/one indicator variable 
for it (Wooldridge, 2000). Values in the column labeled "SSC Lottery Winners" are observed means 
for lottery winners assigned by HSAPS to an SSC. Values in the column labeled "Control Group 
Members" are regression-adjusted means that match the distribution of SSC lottery winners across 
lotteries. In some cases, rounding may cause slight discrepancies when comparing the statistical 
significance of findings in this table (for effects of winning an SSC lottery) with the statistical 
significance of findings in Table 3.7 (for effects of enrolling in an SSC). 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated 
as: ** = 1 percent; * = 5 percent. 

Cohort 1 consists of students in the study who were eighth-graders in the spring of 2005. 
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New York City Small Schools of Choice 
Appendix Table C.l 



Baseline Characteristics of SSC Lottery Participants with Follow-up Data: 
First Year of High School, Cohorts 1 to 4 



Characteristic (%) 


SSC 

Lottery 

Winners 


Control 

Group 

Members 


Estimated 

Difference 


P- Value for 
Estimated 
Difference 


Race/ethnicity 


Hispanic 


46.4 


47.7 


-1.3 


0.251 


Black 


44.9 


43.3 


1.6 


0.149 


Other 


7.1 


7.2 


-0.1 


0.852 


Male 


46.2 


44.8 


1.5 


0.178 


Eligible for free/reduced-price lunch 


84.3 


85.1 


-0.8 


0.391 


English language learner 


6.3 


6.3 


0.0 


0.960 


Special education 3 


5.0 


5.6 


-0.6 


0.223 


Overage for 8th grade b 


20.8 


21.7 


-0.9 


0.337 


8th-grade reading proficiency 3 


Did not meet standards (level 1) 


8.5 


8.4 


0.1 


0.855 


Partially met standards (level 2) 


63.8 


63.9 


-0.2 


0.877 


Fully met standards (level 3) 


21.8 


21.4 


0.4 


0.663 


Met standards with distinction (level 4) 


0.9 


0.7 


0.2 


0.216 


8th-grade math proficiency 3 


Did not meet standards (level 1) 


23.9 


23.4 


0.5 


0.593 


Partially met standards (level 2) 


48.3 


47.7 


0.6 


0.582 


Fully met standards (level 3) 


25.5 


25.7 


-0.1 


0.888 


Met standards with distinction (level 4) 


1.0 


0.9 


0.1 


0.683 


Total number of student observations = 29,81 1 











SOURCES: MDRC's calculations use High School Application Processing System and New York City 
Department of Education (DOE) state test data for eighth-graders from the 2004-2005 to 2007-2008 
school years, as well as data from DOE enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for the 
difference between SSC lottery winners and control group members are obtained from a regression of a 
given baseline characteristic on a series of indicator variables that identify each lottery plus an indicator 
variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter 
indicator variable equals the difference in the mean baseline characteristic for lottery winners and 
control group members. The value for control group members equals the corresponding value for SSC 
lottery winners minus the estimated difference between lottery winners and control group members. To 
facilitate computation, all variables are centered on the mean value for the lottery they represent. This 
approach is equivalent to directly accounting for each lottery by adding a 0/1 indicator variable for it 
(Wooldridge, 2000). In some cases, rounding may cause slight discrepancies. 
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Appendix Table C.l (continued) 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

A chi-square test was used to assess the statistical significance of the overall difference between 
the lottery winners and control group members reflected by the full set of baseline characteristics in 
the table. The resulting chi-square value is not statistically significant (p-value = 0.474). 

Cohorts 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 
2005, 2006, 2007, and 2008, respectively. 

a This sample includes special education students who can be taught in the regular classroom 
setting. Special education students classified by the DOE as requiring collaborative team teaching 
services or self-contained classes are not part of the sample. 

b Lottery participants are classified as "overage for eighth grade" if they were 14 or older on 
September 1 of the eighth-grade school year. 

c Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade 
level for state math and reading exams. Due to missing test scores, the sum of levels 1-4 may not 
add to 100 percent. 
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Baseline Characteristics of SSC Lottery Participants with Follow-up Data: 
Second Year of High School, Cohorts 1 to 3 



Characteristic (%) 


SSC 

Lottery 

Winners 


Control 

Group 

Members 


Estimated 

Difference 


P-Value for 
Estimated 
Difference 


Race/ethnicity 


Hispanic 


46.4 


47.5 


-1.1 


0.357 


Black 


44.8 


43.2 


1.6 


0.151 


Other 


7.3 


7.6 


-0.2 


0.670 


Male 


46.2 


44.4 


1.7 


0.128 


Eligible for free/reduced-price lunch 


84.5 


84.6 


-0.1 


0.943 


English language learner 


6.1 


6.1 


0.0 


0.947 


Special education 3 


4.9 


5.8 


-0.9 


0.116 


Overage for 8th grade b 


19.2 


20.4 


-1.2 


0.230 


8th-grade reading proficiency 3 


Did not meet standards (level 1 ) 


8.3 


8.1 


0.1 


0.830 


Partially met standards (level 2) 


64.1 


64.0 


0.1 


0.930 


Fully met standards (level 3) 


22.0 


21.9 


0.1 


0.904 


Met standards with distinction (level 4) 


0.9 


0.7 


0.2 


0.227 


8th-grade math proficiency 3 


Did not meet standards (level 1 ) 


23.2 


22.9 


0.3 


0.760 


Partially met standards (level 2) 


48.7 


48.4 


0.3 


0.827 


Fully met standards (level 3) 


26.2 


26.0 


0.1 


0.889 


Met standards with distinction (level 4) 


1.1 


1.0 


0.1 


0.738 


Total number of student observations = 21,822 











SOURCES: MDRC's calculations use High School Application Processing System and New York 
City Department of Education (DOE) state test data for eighth-graders from the 2004-2005 to 2006- 
2007 school years, as well as data from DOE enrollment files from the 2005-2006 to 2008-2009 
school years. 

NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for the 
difference between SSC lottery winners and control group members are obtained from a regression 
of a given baseline characteristic on a series of indicator variables that identify each lottery plus an 
indicator variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the 
latter indicator variable equals the difference in the mean baseline characteristic for lottery winners 
and control group members. The value for control group members equals the corresponding value 
for SSC lottery winners minus the estimated difference between lottery winners and control group 
members. To facilitate computation, all variables are centered on the mean value for the lottery they 
represent. This approach is equivalent to directly accounting for each lottery by adding a 0/1 
indicator variable for it (Wooldridge, 2000). In some cases, rounding may cause slight 
discrepancies. 
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Appendix Table C.2 (continued) 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

A chi-square test was used to assess the statistical significance of the overall difference between 
lottery winners and control group members reflected by the full set of baseline characteristics in the 
table. The resulting chi-square value is not statistically significant (p-value = 0.462). 

Cohorts 1, 2, and 3 consist of students in the study who were eighth-graders in the spring of 
2005, 2006, and 2007, respectively. 

a This sample includes special education students who can be taught in the regular classroom 
setting. Special education students classified by the DOE as requiring collaborative team teaching 
services or self-contained classes are not part of the sample. 

b Lottery participants are classified as "overage for eighth grade" if they were 14 or older on 
September 1 of the eighth-grade school year. 

c Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade level 
for state math and reading exams. Due to missing test scores, the sum of levels 1-4 may not add to 
100 percent. 
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Baseline Characteristics of SSC Lottery Participants with Follow-up Data: 
Third Year of High School, Cohorts 1 and 2 



Characteristic (%) 


SSC 

Lottery 

Winners 


Control 

Group 

Members 


Estimated 

Difference 


P-Value for 
Estimated 
Difference 


Race/ethnicity 


Hispanic 


45.9 


47.4 


-1.5 


0.235 


Black 


45.0 


43.2 


1.8 


0.129 


Other 


7.7 


7.7 


-0.1 


0.919 


Male 


46.1 


44.7 


1.4 


0.243 


Eligible for free/reduced-price lunch 


84.9 


84.7 


0.2 


0.850 


English language learner 


6.0 


6.0 


-0.1 


0.921 


Special education 3 


4.9 


5.4 


-0.4 


0.449 


Overage for 8th grade* 3 


16.9 


17.4 


-0.4 


0.653 


8th-grade reading proficiency 13 


Did not meet standards (level 1) 


7.9 


7.6 


0.2 


0.748 


Partially met standards (level 2) 


64.1 


64.2 


0.0 


0.967 


Fully met standards (level 3) 


22.5 


22.5 


0.0 


0.967 


Met standards with distinction (level 4) 


1.0 


0.8 


0.2 


0.292 


8th-grade math proficiency 13 


Did not meet standards (level 1) 


22.1 


22.0 


0.1 


0.942 


Partially met standards (level 2) 


48.9 


48.3 


0.6 


0.638 


Fully met standards (level 3) 


27.1 


27.3 


-0.1 


0.904 


Met standards with distinction (level 4) 


1.1 


1.0 


0.1 


0.821 


Total number of student observations = 13,297 











SOURCES: MDRC's calculations use High School Application Processing System and New York 
City Department of Education (DOE) state test data for eighth-graders from the 2004-2005 and 2005- 
2006 school years, as well as data from DOE enrollment files from the 2005-2006 to 2008-2009 
school years. 

NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for the 
difference between SSC lottery winners and control group members are obtained from a regression of 
a given baseline characteristic on a series of indicator variables that identify each lottery plus an 
indicator variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter 
indicator variable equals the difference in the mean baseline characteristic for lottery winners and 
control group members. The value for control group members equals the corresponding value for SSC 
lottery winners minus the estimated difference between lottery winners and control group members. 
To facilitate computation, all variables are centered on the mean value for the lottery they represent. 
This approach is equivalent to directly accounting for each lottery by adding a 0/1 indicator variable 
for it (Wooldridge, 2000). In some cases, rounding may cause slight discrepancies. 
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Appendix Table C.3 (continued) 



A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

A chi-square test was used to assess the statistical significance of the overall difference between 
lottery winners and control group members reflected by the full set of baseline characteristics in the 
table. The resulting chi-square value is not statistically significant (p-value = 0.599). 

Cohorts 1 and 2 consist of students in the study who were eighth-graders in the spring of 2005 and 
2006, respectively. 

a This sample includes special education students who can be taught in the regular classroom 
setting. Special education students classified by the DOE as requiring collaborative team teaching 
services or self-contained classes are not part of the sample. 

b Lottery participants are classified as "overage for eighth grade" if they were 14 or older on 
September 1 of the eighth-grade school year. 

c Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade level 
for state math and reading exams. Due to missing test scores, the sum of levels 1 -4 may not add to 
100 percent. 
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Baseline Characteristics of SSC Lottery Participants with Follow-up Data: 
Fourth Year of High School, Cohort 1 





SSC 

Lottery 


Control 

Group 


Estimated 


P-Value for 
Estimated 


Characteristic (%) 


Winners 


Members Difference 


Difference 


Race/ethnicity 


Hispanic 


45.0 


46.9 


-1.9 


0.299 


Black 


45.7 


43.3 


2.4 


0.197 


Other 


7.9 


8.4 


-0.5 


0.606 


Male 


44.1 


43.2 


0.9 


0.606 


Eligible for free/reduced-price lunch 


84.0 


85.4 


-1.4 


0.337 


Special education 3 


1.5 


1.1 


0.3 


0.382 


Overage for 8th grade b 


14.3 


12.8 


1.5 


0.284 


8th-grade reading proficiency 0 


Did not meet standards (level 1) 


4.2 


2.7 


1.5 * 


0.030 


Partially met standards (level 2) 


70.1 


70.7 


-0.6 


0.735 


Fully met standards (level 3) 


22.4 


22.8 


-0.4 


0.782 


Met standards with distinction (level 4) 


1.5 


1.1 


0.3 


0.342 


8th-grade math proficiency 0 


Did not meet standards (level 1) 


16.6 


14.9 


1.7 


0.235 


Partially met standards (level 2) 


50.7 


50.9 


-0.2 


0.899 


Fully met standards (level 3) 


30.8 


32.3 


-1.5 


0.393 


Met standards with distinction (level 4) 


1.3 


1.1 


0.2 


0.700 


Total number of student observations = 5,363 



SOURCES: MDRC's calculations use High School Application Processing System and New York 
City Department of Education (DOE) state test data for eighth-graders in the 2004-2005 school year, 
as well as data from DOE enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: Values for SSC lottery winners are the simple means for all lottery winners. Values for the 
difference between SSC lottery winners and control group members are obtained from a regression 
of a given baseline characteristic on a series of indicator variables that identify each lottery plus an 
indicator variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the 
latter indicator variable equals the difference in the mean baseline characteristic for lottery winners 
and control group members. The value for control group members equals the corresponding value for 
SSC lottery winners minus the estimated difference between lottery winners and control group 
members. To facilitate computation, all variables are centered on the mean value for the lottery they 
represent. This approach is equivalent to directly accounting for each lottery by adding a 0/1 
indicator variable for it (Wooldridge, 2000). In some cases, rounding may cause slight discrepancies. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 
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Appendix Table C.4 (continued) 

A chi-square test was used to assess the statistical significance of the overall difference between 
lottery winners and control group members reflected by the full set of baseline characteristics in the 
table. The resulting chi-square value is not statistically significant (p-value = 0.327). 

Cohort 1 consists of students in the study who were eighth-graders in the spring of 2005. 

“This sample includes special education students who can be taught in the regular classroom 
setting. Special education students classified by the DOE as requiring collaborative team teaching 
services or self-contained classes are not part of the sample. 

b Lottery participants are classified as "overage for eighth grade" if they were 14 or older on 
September 1 of the eighth-grade school year. 

“Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade level 
for state math and reading exams. Due to missing test scores, the sum of levels 1-4 may not add to 
100 percent. 
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Appendix D 

Imputing Outcomes for Students 
Who Are Lost Through Attrition 




As noted in Chapter 2, estimated effects of small schools of choice (SSCs) are based on 
administrative data from the New York City Department of Education (DOE). These data are 
only available for students who enroll in a New York City public school. They are not available 
for students who move out of the city, transfer to a private or parochial school, or drop out of 
school. In the present analysis, the percentage of student observations with follow-up data de- 
clines from a high of almost 90 percent in the first year of high school to a low of just under 70 
percent in the fourth year. 

Without follow-up data for all students, SSC effects for the full study sample cannot be 
estimated. However, it is standard procedure to consider how missing data for some students 
might affect results for the full sample. Typically, this is done by assuming values for the miss- 
ing data (“imputing” these values) and repeating the analysis with the assumed values. While 
there are numerous approaches for such imputation, they vary markedly in their complexity, 
transparency, and assumptions, and no consensus exists about which is most appropriate. Thus, 
in the absence of a compelling reason to use a particular approach, the analysis in this report 
uses two simple approaches to illustrate what SSC enrollment effects might possibly be for the 
full study sample. 

The first approach, presented in Appendix Table D.l, makes the relatively conservative 
assumption that any student with missing data for an outcome was unsuccessful on that out- 
come. For example, it is assumed that any student who is missing graduation data has not grad- 
uated. This assumption implies no difference in mean outcomes for SSC lottery winners and 
control group members, which in turn implies zero effect of SSC enrollment. To be more con- 
servative would require assuming that SSCs have a negative effect on graduation rates for stu- 
dents with missing graduation data. Given the sizable positive SSC enrollment effects in earlier 
grades reported in Chapter 3 for the overall sample and many sample subgroups, it seems unrea- 
listically conservative to assume negative effects on subsequent graduation rates for a large sub- 
group such as that comprising students with missing graduation data. 

The second approach to imputing missing outcome data uses information from DOE 
records on student discharge codes to distinguish between students with missing data who are 
indicated as having dropped out of school and all other students for whom data are missing (be- 
cause they left the New York City school district). 1 Table D.2 illustrates the distribution of dis- 
charge codes across dropouts and other categories; the data are shown separately for SSC lottery 
winners and control group members for the fourth year of high school of the first student cohort 
in this study (the only cohort for which four years of high school data are available). Note first 
that differences in all rates of discharge for SSC lottery winners and control group members are 
small and are not statistically significant. Note next that roughly 15 percent of students in the 

^formation on student discharge codes was not used as part of the primary estimation approach for the 
present analysis because this type of information is typically unreliable. 
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New York City Small Schools of Choice 
Appendix Table D.l 

Estimated Effects of SSC Enrollment, Imputing Zeros for Students with 
Missing Follow-up Data for Course Credits: Cohorts 1 to 4 



Outcome 


Target 

SSC 

Enrollees 


Control 

Group Estimated 
Counterparts Effect 


Effect Size 
(Standard 
Deviation) 


P-Value for 
Estimated 
Effect 


Year 1 of hish school (cohorts 1 to 41 

Total credits earned toward graduation 3 
Total number of student observations = 30,959 


10.2 


9.4 


0.8 ** 


0.19 ** 


0.000 


Year 2 of hi 2 h school (cohorts 1 to 31 

Total credits earned toward graduation 3 
Total number of student observations = 26,685 


18.5 


16.4 


2.1 ** 


0.26 ** 


0.002 


Year 3 of hish school (cohorts 1 and 21 

Total credits earned toward graduation 3 
Total number of student observations = 17,984 


24.5 


22.7 


1.8 ** 


0.17 ** 


0.003 


Year 4 of hish school (cohort 11 

Graduated from high school 

Total number of student observations = 8,283 


46.9 


42.5 


4.4 * 




0.015 



SOURCES: MDRC's calculations use High School Application Processing System data from eighth-graders in the 
2004-2005 to 2007-2008 school years, as well as data from New York City Department of Education attendance, 
course credits, Regents exam, transactional, and enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: Appendix A describes how values in the column labeled "Target SSC Enrollees" are estimated. 
Appendix A also describes how values in the column labeled "Estimated Effect" are estimated. Values in the 
column labeled "Control Group Counterparts" are differences between corresponding values in the first and third 
columns. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated as: ** = 1 
percent; * = 5 percent. 

The estimated effect size for each measure is calculated as a proportion of the standard deviation of the 
outcome for control group counterparts. 

Cohort 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 2005, 2006, 
2007, and 2008, respectively. 

a The "total credits earned toward graduation" measure is the aggregate number of course credits earned toward 
fulfilling the New York State graduation requirements. The credit requirements are as follows: 31 core subject 
credits, including 8 credits each of English and social studies; 6 credits each of math and science; 2 credits of arts; 
1 credit of health; and 13 additional credits, including 4 credits ofphysical education, 2 credits of a foreign 
language, and 7 credits of electives. 
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New York City Small Schools of Choice 
Appendix Table D.2 



Estimated Effects of Winning an SSC Lottery on DOE-Defined Categories 
of Discharge Status: Fourth Year of High School, Cohort 1 



Outcome 


SSC 

Lottery 

Winners 


Control 

Group Estimated 
Members Effect 


P-Value for 
Estimated 
Effect 


Students missing outcome data in 4th year 










Dropped out 


14.6 


15.6 


-1.0 


0.556 


Transferred to an alternative program 


10.5 


11.8 


-1.3 


0.395 


Other 3 


7.3 


7.5 


-0.2 


0.838 


Students not missing outcome data in 4th year 


68.3 


69.4 


-1.1 


0.398 


Total number of student observations = 8,283 



SOURCES: MDRC's calculations use High School Application Processing System data from 
eighth-graders in the 2004-2005 to 2007-2008 school years, as well as data from New York City 
Department of Education (DOE) course credits, transactional, and enrollment fdes from the 2005- 
2006 to 2008-2009 school years. 

NOTES: This table presents the estimated effects for students who have follow-up course credits 
data. Estimated differences between SSC lottery winners and control group members are 
regression-adjusted as described by Equation A.l in Appendix A. Variables in the regression are 
measured as deviations from their lottery mean in order to account for the lottery for each sample 
point. This approach is equivalent to directly accounting for each lottery by adding a 0/1 indicator 
variable for it (Wooldridge, 2000). Values in the column labeled "SSC Lottery Winners" are 
observed means for lottery winners assigned by HSAPS to an SSC. Values in the column labeled 
"Control Group Members" are regression-adjusted means that match the distribution of SSC lottery 
winners across lotteries. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 

Cohort 1 consists of students in the study who were eighth-graders in the spring of 2008. 

a The "Other" category consists of students who transfer to a school outside the DOE system, 
enter into an early college program, or are reported as deceased. For accountability purposes, these 
students are are not included in the publicly reported graduation cohort. 



first study cohort are coded as dropouts by the time they reach their fourth year of high school. 
The remaining approximately 1 8 percent of students are coded as having missing data for some 
other reason. 

Using the information about school dropouts, the second approach imputes a value 
of zero for outcomes for students who are coded as high school dropouts. For the remaining 
students with missing data, values are imputed based on the distribution of values observed 
for students whose data are not missing. This is accomplished by taking the mean and stan- 
dard deviation of the outcome for control group members with data and assuming a normal 
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Appendix Table D.3 

Estimated Effects of SSC Enrollment, Imputing Zeros for Dropouts and Values 
in Accord with Control Group Distribution for All Other Students 
with Missing Follow-up Data for Course Credits: Cohorts 1 to 4 



Outcome 


Target 

SSC 

Enrollees 


Control 

Group Estimated 
Counterparts Effect 


Effect Size 
(Standard 
Deviation) 


P-Value for 
Estimated 
Effect 


Year 1 of hi 2 h school (cohorts 1 to 4) 

Total credits earned toward graduation" 1 
Total number of student observations = 30,959 


10.6 


9.7 


0.9 ** 


0.21 ** 


0.000 


Year 2 of hish school (cohorts 1 to 31 

Total credits earned toward graduation 3 
Total number of student observations = 26,685 


20.2 


18.1 


2.1 ** 


0.25 ** 


0.001 


Year 3 of hi 2 h school (cohorts 1 and 21 

Total credits earned toward graduation 3 
Total number of student observations = 17,984 


27.4 


25.4 


2.0 ** 


0.18 ** 


0.003 


Year 4 of hish school (cohort 11 

Graduated from high school 

Total number of student observations = 8,283 


55.8 


51.2 


4.6 * 




0.012 



SOURCES: MDRC's calculations use High School Application Processing System data from eighth-graders in the 
2004-2005 to 2007-2008 school years, as well as data from New York City Department of Education attendance, 
course credits, Regents exam, transactional, and enrollment files from the 2005-2006 to 2008-2009 school years. 



NOTES: Appendix A describes how values in the column labeled "Target SSC Enrollees" are estimated. 
Appendix A also describes how values in the column labeled "Estimated Effect" are estimated. Values in the 
column labeled "Control Group Counterparts" are differences between corresponding values in the first and third 
columns. 

A two-tailed t-test was applied to the estimated effect. Statistical significance levels are indicated as: ** = 1 
percent; * = 5 percent. 

The estimated effect size for each measure is calculated as a proportion of the standard deviation of the 
outcome for control group counterparts. 

Cohort 1, 2, 3, and 4 consist of students in the study who were eighth-graders in the spring of 2005, 2006, 
2007, and 2008, respectively. 

a The "total credits earned toward graduation" measure is the aggregate number of course credits earned toward 
fulfilling the New York State graduation requirements. The credit requirements are as follows: 31 core subject 
credits, including 8 credits each of English and social studies; 6 credits each of math and science; 2 credits of arts; 
1 credit of health; and 13 additional credits, including 4 credits of physical education, 2 credits of a foreign 
language, and 7 credits of electives. 
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distribution of these values. This part of the imputation (like the first imputation approach) 
makes the relatively conservative assumption that SSCs had no effect on graduation rates 
for students with missing follow-up data (because it imputes the same mean value of the 
outcome for SSC lottery winners and control group members who are not dropouts but have 
missing data). 

Table D.3 presents results using the second imputation approach corresponding to those 
in Table D. 1 for the first imputation approach. As expected, estimated SSC enrollment effects in 
Table D.3 are essentially the same as those in Table D. 1 . 
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Appendix E 

Sources and Description of Data Collected 

for the Study 




New York City Small Schools of Choice 
Appendix Table E.l 



Sources and Description of Data Collected for the Study 



Data Source 


Sample 


School Years Used 


Description of Data 


High School Application 
Processing System 
(HSAPS) 


Data are available for all 
eighth-grade students who 
completed their application to 
the High School Application 
Processing System. 


2004-2005 to 2007-2008 


These files contain information about the school 
choices and rankings made by each student, as well 
as a student’s geographic priority and “known” 
status for each school. The files also contain the 
program/school to which each student was 
assigned by the HSAPS algorithm as well as their 
eighth-grade demographic characteristics. 


New York City 
Department of Education 
(DOE) October enrollment 
data 


Data are available for 6th- to 
12th-graders enrolled in public 
schools. This file contains all 
students enrolled in grades 6-12 
as of October of each school 
year. 


2005-2006 to 2008-2009 


The October enrollment files contain demographic 
and identification characteristics for each student. 
These data are used to determine where each 
student enrolls at the beginning of the school year. 


DOE June enrollment data 


Data are available for 6th- to 
12th-graders enrolled in public 
schools. This file contains all 
students who have enrolled in 
grades 6-12 at any point during 
the school year. 


2005-2006 to 2008-2009 


The June enrollment files contain demographic and 
identification information for each student as of the 
end of the school year. These data also provide 
detailed information about each student’s 
disposition at the end of the year. 


DOE Regents exam data 


Data are available for all 
students who attempted a 
Regents exam. Data are 
collected at each Regents exam 
testing date, and compiled by 
school year. 


2005-2006 to 2008-2009 


These data contain information on test subject, test 
date, and total score for each Regents exam taken 
in the school year. 


DOE attendance data 


Data are available for all 
students enrolled in the New 
York City public school system. 
Data are collected at the end of 
each school year. 


2005-2006 to 2008-2009 


The attendance files contain measures of days 
present, days absent, and days enrolled in the New 
York City public school system. This information 
was used to construct the attendance rate measures. 



(continued) 
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Appendix Table E.l (continued) 



Data Source 


Sample 


School Years Used 


Description of Data 


DOE course credits data 


Data are available for all high 
school students in the New 
York City public school system. 
Data are collected at the end of 
each school year, and include 
credits accumulated during 
summer school. 


2005-2006 to 2008-2009 


The course credits tiles contain the number of 
credits attempted, passed, and failed in each of the 
nine DOE classified subject areas: English, math, 
social studies, science, foreign language, 
business/technology, arts, physical education, and 
miscellaneous. 


DOE student transactional 
file 


Data are available for all high 
school students in the New 
York City public school system. 
Data are collected throughout 
the school year. 


2005-2006 to 2008-2009 


The transactional tile records each movement a 
student makes into or out of a school in the DOE 
system. The final transaction a student made on 
this file was used to construct the graduation 
measures for this study. 


DOE state test data 


Data are available for all 
students in grades 3-8 who took 
the New York State reading or 
math tests. 


2003-2004 to 2008-2009 


The state test files contain the scale score and 
performance level for each student who took the 
New York State mathematics and reading tests. 


New York State Report 
Card 


Data collected from the New 
York State Education 
Department (NYSED) for all 
New York City high schools. 


2002-2003 to 2007-2008 


Data include characteristics of the school, such as 
student body demographics, school location, 
student enrollment, and teacher experience and 
tenure. 


Administrative records 
provided by the Gates 
Foundation and 
intermediaries 


Data are available for all new 
small schools that started after 
the 2002-2003 school year and 
received funding from the Bill 
& Melinda Gates Foundation. 


2002-2003 to 2008-2009 


These school-level data contain information on 
whether a new small school was started with 
funding from the Bill & Melinda Gates 
Foundation. 


DOE school-level 
administrative records 


Data on new small schools, new 
small learning communities, 
and new career and technical 
education academies opened as 
of the 2002-2003 school year 
were provided by the New York 
City DOE. 


2002-2003 to 2008-2009 


These school-level data contain information on 
new small school characteristics, small learning 
community characteristics, and career and technical 
education academy characteristics. Examples of 
these characteristics include school/program name, 
location, school opening date, and expected grade 
configuration. 





Appendix F 

Variation in the Fourth-Year Effects 




New York City Small Schools of Choice 
Appendix Table F.l 

Variation in Effects of SSC Enrollment on Graduation, 
by Student Characteristic: Fourth Year of High School, Cohort 1 



Student Characteristic 


Estimated 

Effect 


P- Value for 
Estimated 
Effect 


8th-srade readme DrofIciencv a 






Did not meet standards (level 1) 


-9.8 


0.413 


Partially met standards (level 2) 


94 ** 


0.002 


Fully met standards (level 3) 


9.8 


0.062 


Met standards with distinction (level 4) 


- 


- 


8th-srade math DrofIciencv a 






Did not meet standards (level 1) 


9.0 


0.131 


Partially met standards (level 2) 


7.2 * 


0.046 


Fully met standards (level 3) 


6.2 


0.156 


Met standards with distinction (level 4) 


- 


- 


Low-income status 






Eligible for free/reduced-price lunch 


7.0 * 


0.032 


Not eligible for free/reduced-price lunch 


4.7 


0.235 


Race/ethnicitv. bv sender 






Black male 


6.9 


0.198 


Hispanic male 


0.4 


0.938 


Other male 


10.9 


0.358 


Black female 


2.3 


0.604 


Hispanic female 


2.4 


0.669 


Other female 


6.4 


0.603 


Choice level (of 12) at which enrollee DarticiDated in lottery 






1 st choice 


7.0 * 


0.038 


2nd-3rd choices 


8.3 


0.055 


All other choices 


8.3 


0.160 



(continued) 
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Appendix Table F.l (continued) 

SOURCES: MDRC's calculations use High School Application Processing System and New York City 
Department of Education (DOE) state test data from eighth-graders in 2004-2005, as well as data from DOE 
course credits, transactional, and enrollment files from the 2005-2006 to 2008-2009 school years. 

NOTES: This table presents the estimated effects for students who have follow-up course credits data. Each 
panel in this table divides students into subgroups based on a given characteristic. Within each subgroup a two- 
tailed t-test was applied to the estimated effect, with statistical significance levels indicated as: ** = I percent; * 
= 5 percent. An F-test was used to assess the statistical significance between subgroups; none of the subgroup 
differences in this table is statistically significant. 

Cohort 1 consists of students in the study who were eighth-graders in the spring of 2005. 

Estimated effects on level 4 math and reading proficiency are not estimable due to small sample sizes. 
a Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade level for state 
math and reading exams. 
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Appendix G 

Small Schools of Choice in This Study 




This report provides findings on the effects of New York City’s “small schools of choice” 
(SSCs). SSCs are technically defined as: 

• High schools (as opposed to middle/high schools or other school types), which are 
intended to serve only grades 9 through 12 at capacity 1 

• Schools that were founded in 2002 or later 2 

• Schools that employed the “limited unscreened” selection method in the High 
School Application Processing System (HSAPS) 3 

Additionally, because this study utilizes naturally occurring lotteries that take place 
within HSAPS when an SSC has more available applicants than seats, only those SSCs that 
were also oversubscribed serve as the SSCs that generate the study sample. 4 Hie 105 study 
SSCs are listed below by borough. 

All schools meeting the aforementioned criteria are additionally small — serving 550 
students or fewer. 

Brooklyn 

Academy for Environmental Leadership 

Academy for Y oung Writers 

Academy of Hospitality and Tourism 

Academy of Urban Planning 

Arts and Media Preparatory Academy 

Brooklyn Academy for Science and the Environment 

Brooklyn Community High School of Communication, Arts and Media 

Brooklyn Preparatory High School 

Bushwick Leaders’ High School for Academic Excellence 



1 Three of the schools in the study sample, while founded as high schools, began serving sixth- to eighth- 
graders during the study period. 

2 The study period includes academic years 2005-2006 through 2008-2009. Thus, only schools that were 
founded as of September 2008 are characterized as SSCs. 

’Limited unscreened schools do not impose academic requirements but instead give preference to students 
who live within a certain geographic area and have attended a school’s open house or the school’s booth at a 
school fair, or who are otherwise “known” to the school. 

4 Appendix A provides a detailed description of how HSAPS generates lotteries. In order to be 
oversubscribed for lottery purposes, an SSC must have more available applicants (who have not already been 
matched to one of their prior school choices) than seats. Thus, some schools that appear to have been 
oversubscribed (for example, a school described in the New York City Department of Education High School 
Handbook as having had 500 “total applicants” for 108 “program seats”) may not have actually been 
oversubscribed for the purposes of generating a lottery. 
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Bushwick School for Social Justice 

Expeditionary Learning School for Community Leaders 

FDNY High School for Fire and Life Safety 

Foundations Academy 

Frederick Douglass Academy VII 

Green School 

High School for Civil Rights 

High School for Global Citizenship 

High School for Service and Learning 

High School for Youth and Community Development 

High School of Sports Management 

International Arts Business School 

It Takes A Village Academy 

New York Harbor School 

Performing Arts and Technology High School 

Rachel Carson High School of Coastal Studies 

Urban Assembly School for Law and Justice 

Urban Assembly School of Music and Art 

Victory Collegiate High School 

Williamsburg High School for Architecture and Design 

Williamsburg Preparatory School 

World Academy for Total Community Health 



Manhattan 

Essex Street Academy 

Facing History School 

Food and Finance High School 

Henry Street School for International Studies 

High School for Arts, Imagination, and Inquiry 

High School of Hospitality Management 

James Baldwin School 

Lower Manhattan Arts Academy 

Manhattan Theatre Lab 

Mott Hall High School 

Pace High School 

Urban Assembly Academy of Government and Law 
Urban Assembly School for Media Studies 
Urban Assembly School for the Performing Arts 
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Urban Assembly School of Business for Young Women 
Urban Assembly School of Design and C onstruction 



Queens 

Academy of Finance and Enterprise 
Cypress Hills Collegiate Preparatory School 
East- West School of International Studies 
Excelsior Preparatory High School 
Frederick Douglass Academy VI 
High School of Applied C ommunication 
Pan American International High School 
Queens Preparatory Academy 



Staten Island 

College of Staten Island High School for International Studies 



Bronx 

Academy for Language and Technology 
Astor Collegiate Academy 
Bronx Academy of Health Careers 
Bronx Academy of Letters 
Bronx Aerospace Academy 
Bronx Center for Science and Mathematics 
Bronx Engineering and Technology Academy 
Bronx Expeditionary Learning High School 
Bronx Guild 

Bronx Health Sciences High School 

Bronx High School for the Visual Arts 

Bronx High School for Writing and Communication Arts 

Bronx High School of Performance and Stagecraft 

Bronx Lab School 

Bronx Leadership Academy II 

Bronx School of Law and Finance 

Bronx Theatre High School 

Collegiate Institute for Math and Science 
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Community School for Social Justice 
Discovery High School 
Dreamyard Preparatory School 
Eagle Academy for Y oung Men 
Explorations Academy 
Felisa Rincon De Gautier Institute 

Gateway School of Environmental Research and Technology 

Global Enterprise Academy 

High School for Contemporary Arts 

High School for Teaching and the Professions 

High School for Violin and Dance 

High School of Computers and Technology 

Holocombe L. Rucker School of Community Research 

Knowledge and Power Preparatory Academy International High School 

Leadership Institute 

Metropolitan High School 

Millennium Art Academy 

Morris Academy for Collaborative Studies 

Mott Hall Bronx High School 

Mott Haven Village Preparatory High School 

New Explorers High School 

Pablo Neruda Academy for Architecture and World Studies 
Peace and Diversity Academy 
Pelham Preparatory Academy 

Renaissance High School of Musical Theater and Technology 

School for Community Research and Learning 

School for Excellence 

Sports Professions High School 

Urban Assembly Academy for Careers in Sports 

Urban Assembly Academy for History and Citizenship for Young Men 

Validus Preparatory Academy School for Excellence 
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Appendix H 

New York City Department of Education’s 
New Secondary School Application, 2008 




New York City Department of Education 



NEW SECONDARY SCHOOL 
APPLICATION 
2008 EDITION 



APPLICATION FOR GRADES 6-8, 6-12 OR 9-12 



Office of Portfolio Development 
52 Chambers Street 
New York, New York 10007 
Telephone: 212.374.2371; Facsimile: 212.374.5581 
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Introduction 



The New York City Department of Education (DOE) Office of Portfolio Development (OPD) is 
pleased to present the new school application process for opening DOE schools in September 
2008. The development of excellent new small schools is a key component of the Department 
of Education’s second term Children First reform agenda and an important strategy for focusing 
attention on the vision of the future through demonstrating what is possible in public education. 
The Office of Portfolio Development seeks to develop a diverse portfolio of new schools that 
incorporate knowledge from research and from expert practice about the critical elements of 
what makes an effective school. 

Completing this application is a rigorous process that will require applicants to form planning 
teams, work together closely over time, and develop and revise a portfolio of documents. The 
documents that each planning team creates are meant to be ones that could be used in the new 
school and should reflect the planning team’s fundamental beliefs about education. 

To support the efforts of the planning team, the Office of Portfolio Development has designed a 
series of professional development workshops. These sessions will review all the major 
aspects of creating a school and provide technical assistance as applicants develop the most 
thoughtful and effective plans possible for starting a school. The calendar below is a tentative 
list of professional development workshops. Please check our website 
http://schools.nyc.gov/NewSchools to confirm workshop dates, times, and locations. 



The Application Process 


Description 


Spring/Summer: Introductions, Coalition- 

Building, and Getting Started 

• Open House for 2008 New Schools 
Thursday April 26 

• Information Sessions 
May 10 - charter schools 
May 17 - transfer schools 
May 24 - ELL focused schools 
May 31 - CTE focused schools 

• Summer Professional Development Workshops 
July 10, 12,17, 19 


Individuals, including potential principal and teachers, 
intermediaries, community based organizations, and other 
groups interested in developing new schools come 
together to share ideas and explore possible 
collaborations. 

New school development workshops on curriculum & 
assessment design, school culture & personalization, 
differentiated instruction and staffing and scheduling. 
Times and location TBD - see website for details: 
http://schools.nvc.aov/NewSchools 


Fall: Professional Development, Executive 
Summary and Final Application Submissions 

• Fall New School Kick-Off 
Wednesday September 26 

• Executive Summary due 
October 19, 2007 

• Professional Development Workshops 
October 3, 10, 17, 27 November 7, 14 December 
5 

• Final New School Application due 
November 28, 2007 


Professional development workshops will support 
planning teams as they work on the core issues of school 
creation, ranging from curriculum and assessment to 
graduation criteria, school culture to hiring. All teams are 
encouraged to attend. See website for details: 
http://schools.nvc.aov/NewSchools 

The executive summary and final application should draw 
from these sessions. 
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There are three distinct stages to the 2008 new school application process: 

Stage 1 Executive Summary. This first step is a short but critical step in the application 
process. All applicants must submit an Executive Summary. All applicants may 
continue on to submit the application portfolio - no feedback on the Executive 
Summary will be provided. The Executive Summary will include biographical 
information, a vision statement, and resumes of planning team members. This 
must be emailed to: NewSchoolsTeam@schools.nyc.gov by close of business 
October 19, 2007 . 

Stage 2 Final Application Portfolio. Based on the “Elements of Effective Schools” (listed 
herein), application proposals should reflect the essence of what each planning 
team believes will lead to the development of a successful school. 

Email completed Application Portfolio to: NewSchoolsTeam@schools.nyc.gov by 
close of business Wednesday November 28, 2007. 

Stage 3 Interviews will be conducted during the week of December 10-14, 2007. Only 

teams with the most developed application portfolios will be invited to interview with 
a panel assembled by the Office of Portfolio Development. The interview will 
provide the planning team with an opportunity to answer questions about the 
application and demonstrate its quality. 

Chancellor approval of new schools opening in September 2008 will take place in late January 

2008. 

For leaders of approved schools, the New York City Leadership Academy sponsors the New 

School Intensive from February through August 2008, and then through the first year of the 

school’s implementation. 
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Stage 1 : The Executive Summary, Due October 19, 2007 
Part A: Applicant Information 

1 . Name of Proposed School 

2. Desired Location of Proposed School (Please note that schools are located where there is 
capacity and demonstrated educational need. The first priority is to replace schools 
selected to close due to historical under-performance. New school applications that reflect 
the intersection of capacity and need will be prioritized.) 

3. Name of Proposed School Leader 

■ Mailing address 

■ Telephone number(s) - specify work, home, cell 

■ Facsimile number 

■ E-mail address 

■ Leadership Program (if applicable, ex. New Leaders for New Schools, NYC 
Leadership Academy, etc.) 

4. Name of Intermediary Organization/Lead Partner, if applicable 

■ Mailing address 

■ Contact person 

■ Telephone number(s) of contact person 

■ E-mail address of contact person 

5. Proposed grade level(s) to be served in the first year 

6. Proposed grade levels to be served at scale (full capacity) 

7. List members of the planning team. Include their affiliations and/or potential roles at the school. 
The planning team should include teachers, parents, students, intermediary representatives (if 
applicable), community-based partner representatives (if applicable) and the designated 
leadership. 

8. School Theme. Check one (if applicable): 

□ Architecture & Urban Planning □ Business, Finance, & Entrepreneurship 

□ Health & Medicine □ Humanities & Classical Studies 

□ Literature, Writing, & Communication □ Multicultural/Multilingual Education 

□ Performing & Visual Arts □ Science, Math, & Technology 

□ Service, Leadership, Law & Justice □ Other: 
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PART B: Executive Summary (No More Than Four Pages) 



Articulate the overarching mission of the proposed school and the components of the school 
design and support structures that are most essential to achieving that mission. Briefly state the 
reasons for opening this school in the community identified (if applicable) and the ways in which 
the school will benefit students. Include a description of the unique educational experiences of 
students in the proposed new school. To demonstrate the school’s mission, include a one page 
task that students will be expected to complete by the end of the first year. 



Part C: Evidence of Capacity 

Please submit the resumes and partnership agreements to the fullest extent possible as 
part of the Executive Summary. 

1 . Planning Team Capacity: Write a one page overview of the strengths of the planning team 
that serves as an introduction to the resumes. Provide a resume for each member of the 
planning team, including his/her educational and employment history. The resume and/or 
cover letter should describe the leadership capacity of the key personnel, including evidence 
of expertise and a strong track record in the following areas: 

a. Instructional leadership, including 

■ Organizing toward high student achievement 

■ Professional development 

■ Curriculum development 

■ Youth development 

b. Urban school development and operation 

c. Parent and community engagement 

d. Financial management 

e. Creation of strategic partnerships 

2. Institutional Partnerships: 

a. If the proposed school anticipates partnering with an outside entity (i.e. an intermediary 
and/or community-based organization), provide evidence of the organization’s track 
record in terms of a. - e. above. 

b. Describe the role that the partner(s) would play in the school’s educational plan and 
operation. 

3. Leadership Capacity: 

a. Attach a copy of the proposed school leader’s SAS/SBL or SDA/SDL certificate or proof 
of enrollment in a New York State principal certification program, indicating license date. 

b. Attach the names, positions and contact information of three potential referees who will 
testify to the integrity and leadership qualities of the proposed school leader. 
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Stage 2: The Application Portfolio, Due November 28, 2007 



The New School Application is a portfolio of documents that form the core of a school’s vision. 
Each document is an authentic component toward building a school, and taken together they 
will be used to assess the viability of the plan. The portfolio is a window into the school’s design 
and is not intended to be a comprehensive blueprint. View each document as part of the whole. 
Please adhere to the 30 page limit for Part A & Part B (excluding resumes). Being succinct is 
critical for the audiences of each component of the application. Each part of the portfolio will 
demonstrate the team’s capacity to execute the school’s mission. 

The Office of Portfolio Development will evaluate new school applications using research-based 
evidence of the following Elements of Effective Schools : 

1 . Strong leadership and a mission that teachers, administrators, and students know and 
support. 

2. A structure, including elements such as reduced teacher load that ensures that students 
will be known well by their teachers and other school staff. 

3. A small team of qualified teachers responsible for a manageable number of students for 
at least a full school year that has the autonomy necessary to determine what students 
learn and how and what they need to make regular progress towards graduation. 

4. High expectations for all students and a standards-based, academically rigorous 
curriculum that connects what students learn with college and career goals. 

5. Performance-based assessments for students and teachers and a culture of continuous 
improvement and accountability for student success. 

6. A structure that fosters the development of authentic, sustained, caring, and respectful 
relationships between teachers and students and among staff members. Advisories of 
10-15 students are one strategy to achieve this goal. 

7. A school schedule that includes longer instructional blocks that promote interdisciplinary 
work, teacher collaboration, and reduced student loads. This schedule should be 
coupled with collaborative team planning and professional development time within the 
regular school schedule so that teachers can form a professional community. 

8. A well-defined plan to service the learning needs of the full range of students in the 
community, including special education students and English Language Learners. 

9. Connections between what students learn in school to their lives and communities 
through internships, mentoring experiences, and service learning opportunities. 

10. Partnerships with students, parents, and community organizations and institutions as 
key collaborators and stakeholders. 
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As part of Children First, new schools created in 2008 are designed specifically to meet the 
needs of under-served communities. Applications should clearly state how all students, 
including those who are performing below grade level, students entitled to special education 
services, and English language learners will be successful in this new school design. The 



following data is a potential snapshot of the partial incoming class - applicants are advised to 



specifically refer to this data while designing their new school. 

The following hypothetical data table reflects a sample of the students who might attend the new 
school. The data set is selected from previous school year reports. Use this sample data to 
inform the new school application. 


Student 

ID 


Sex 


SC or 

ELL SETSS CTT 


Title 1 
Eligible 


Days 

Absent 


ELA 

Score 


Math 

Score 


1 


M 


Y 


Y 


15 


1 


1 


2 


M 






0 


3 


2 


3 


M 


Y Y 


Y 


7 


2 


1 


4 


M 






16 


2 


1 


5 


M 




Y 


33 


2 


1 


6 


M 






1 


1 


2 


7 


M 




Y 


8 


2 


2 


8 


M 






7 


2 


1 


9 


F 


Y 




8 


2 


2 


10 


F 




Y 


28 


4 


4 


11 


F 




Y 


4 


1 


1 


12 


M 






14 


2 


2 


13 


M 




Y 


0 


4 


3 


14 


M 






10 


1 


2 


15 


F 






49 


2 


3 


16 


F 


Y 


Y 


2 


2 


1 


17 


F 




Y 


20 


2 


3 


18 


F 




Y 


15 


3 


2 


19 


F 


Y 


Y 


2 


1 


2 


20 


F 




Y 


16 


2 


2 


21 


F 




Y 


22 


3 


4 


22 


M 






19 


1 


2 


23 


M 




Y 


4 


2 


1 


24 


F 


Y 


Y 


0 


1 


1 


25 


F 




Y 


12 


2 


2 


26 


M 




Y 


17 


2 


2 


27 


M 




Y 


20 


2 


3 


28 


M 




Y 


0 


3 


2 


29 


M 


Y 




9 


2 


1 


30 


F 


Y 


Y 


14 


1 


2 
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THE APPLICATION PORTFOLIO 



Part A: Applicant Information 

1 . Name of Proposed School 

2. Desired Location of Proposed School (Please note that schools are located where there is 
capacity and demonstrated educational need. The first priority is to replace schools 
selected to close due to historical under-performance. New school applications that reflect 
the intersection of capacity and need will be prioritized.) 

3. Name of Proposed School Leader 

■ Mailing address 

■ Telephone number(s) - specify work, home, cell 

■ Facsimile number 

■ E-mail address 

■ Leadership Program (if applicable, ex. New Leaders for New Schools, NYC 
Leadership Academy, etc.) 

4. Name of Intermediary Organization/Lead Partner, if applicable 

■ Mailing address 

■ Contact person 

■ Telephone number(s) of contact person 

■ E-mail address of contact person 

5. Proposed grade level(s) to be served in the first year 

6. Proposed grade levels to be served at scale (full capacity) 

7. List members of the planning team. Include their affiliations and/or potential roles at the school. 
The planning team should include teachers, parents, students, intermediary representatives (if 
applicable), community-based partner representatives (if applicable) and the designated 
leadership. 

8. School theme. Check one (if applicable): 

□ Architecture & Urban Planning □ Business, Finance, & Entrepreneurship 

□ Health & Medicine □ Humanities & Classical Studies 

□ Literature, Writing, & Communication □ Multicultural/Multilingual Education 

□ Performing & Visual Arts □ Science, Math, & Technology 

□ Service, Leadership, Law & Justice □ Other: 
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THE APPLICATION PORTFOLIO 



Part B: Vision & Values 

1. Introduce the application portfolio with a one page cover letter that sets the 
documents in context and frames the school’s vision. A possible way to frame the 
introduction is to address these questions: 

■ What are your school’s core values? 

■ How are these values reflected in the elements of this portfolio? 

■ How have these values been reflected in the team’s process of developing this new 
school proposal? 

2. The second set of documents orients prospective students and their families to the 
vision and values of the new school. These include: 

a. School Directory Page. As a component of the school admissions process, complete 

the attached page - labeled as Attachment 2A (page 15). ‘In Your Own Words’ presents 
an opportunity to explain the key features of this school to parents and students - it must 
be only one paragraph, free of educational jargon. The other parts of the page provide a 
snapshot of the school program in its first year. 

b. Introductory letter. A welcome letter to parents and accepted students can include the 

vision of the school as well as how this will become reality. It should include discussion 
of the specific role of Intermediary or community based partner. If necessary, this letter 
should specifically welcome the target student population your school is designed to 
serve (for example, English language learners.) 

c. Sample outreach. An example of how you will attract students to your new school. 

Please note that all work in this application should reflect your understanding of the 
students in your school, including under-prepared and low-performing students. At full 
capacity, new schools may also serve 15% students entitled to special education 
services and 20% English language learners. 

As you recruit students and parents, communicate your vision and describe the new 
culture in one sample outreach material you might provide: a brochure to distribute at 
recruitment fairs, talking points at a Community Education Council meeting, a 3 minute 
DVD that shows the daily life of a student in the school, etc. 

3. The third set of documents orients students and their families to the culture of the 
school. These include: 

a. The introduction to the student handbook that frames the culture of your school 

b. A sample document that will explain the rules and consequences for students - or a 
portion of the rules 

c. A sample of a student’s weekly and daily schedule 

d. An explanation - written for students - of the performance expectations (qualitative 
and quantitative) for students graduating from the school, beyond the state mandated 
examinations 
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4. The fourth set of documents orients teachers to the Curriculum and Instruction 

model, and includes the following: 

a. Curriculum Scope and Sequence: Show the scope and sequence of subject 
courses that defines the curriculum across all the grades the school will serve at full 
capacity. Present this in chart form by course title only. Include courses that highlight 
the unique theme and mission of the school. It is not necessary to describe the 
courses. 

b. Curriculum Map: Identify one subject or course area and briefly describe the 
curriculum scope and sequence for each grade level of the school at full capacity. 

Pick the subject area which is central to the theme of the school. For each year, 
include a 2-3 sentence description of critical components of the curriculum and 
indicate briefly what the students will know and be able to do at the end of each year. 
This curriculum map may include, but is not limited to: Essential Questions, Skills and 
Content Knowledge, Major Projects. This should be presented in the form of a chart. 

c. Sample Unit and Lesson Plan: Provide a sample unit and a sample daily lesson 
plan from the first year curriculum described in the Curriculum Map above. The 
sample unit and lesson plan should represent the kind of teaching and learning you 
would like to see in the school. Be explicit in the unit and lesson plan as to what 
supports will be provided for students entitled to special education services, English 
language learners and/or students who scored Level 1 or 2 on promotional tests. For 
schools proposing grades 6-12 at full capacity, include a sample unit and lesson plan 
from both the 6 th and 9 th grades. 

d. Second Sample Unit and Lesson Plan: Provide a second sample unit and a lesson 
or project plan from another subject area in the same year described above. This 
second sample should demonstrate how literacy or numeracy is taught across the 
curriculum. Be explicit in the unit and lesson plan as to what supports will be provided 
for students entitled to special education services, English language learners and/or 
students who scored Level 1 or 2 on promotional tests. For schools proposing grades 
6-12 at full capacity, include a second sample unit and lesson plan from both the 6 th 
and 9 th grades. 

e. Assessment: Design an assessment (major project, test, performance, etc) and a 
rubric that will show students’ level of mastery on the task described in the Sample 
Unit (c) above. The assessment should be reflective of the kind of task that is central 
to the theme or approach to learning in the school. Be explicit as to how the 
assessment may be modified to measure the progress of students entitled to special 
education services, English language learners and/or students who scored Level 1 or 
2 on promotional tests. 

f. Professional Development: Design a professional development session that enables 
teachers to become better skilled at implementing the kind of lesson and teaching 
strategies described in the Sample Unit (c) above. 
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5. The fifth set of documents focus on the school’s vision and values with regard to 
teacher performance: 



a. Hiring Criteria: Write a job description for a teacher in the school and the criteria that 
a fully qualified candidate should meet 

b. Weekly Teacher Schedule: Provide a sample for any one subject area teacher 

c. Professional Development Plans: Prioritize two topics for professional 

development during the summer and first year for teachers in the new school. For 
each, provide a sample of a professional development lesson plan that is in-house 
and designed by the instructional leader. Topics may include, but are not limited to: 

■ Developing curriculum 

■ Developing school culture 

■ Serving the needs of low-performing students, students entitled to special education 
services, English language learners 

■ Using data to build a culture of continuous learning 

■ Rules, regulations, and discipline code 

■ Parent and community engagement 

d. Using data to build a culture of continuous learning. Using the sample data listed 
on page 6: 

■ How will you use this data as a diagnostic over the summer? 

■ How will you work with teachers over the summer to enable them to make sense of 
the data? Design a professional development session for teachers using this data. 



5. The sixth set of documents describes the leadership and accountability systems 
of the school: 

a. Accountability: Describe one or two systems the school will put in place to evaluate 

results from periodic and formative assessments, and to approach the Quality 
Review benchmarks, as well as parent, teacher, and student satisfaction surveys. 

b. Governance: Beyond the mandated School Leadership Team and Parent 

Association, what is one critical committee you plan to create in the school’s first 
year? Who will be on this committee? How will the committee members be 
selected? What decisions will members have the authority to make? Explain your 
choice in one page. 

c. Budget & Staffing Plan: Using the budget model provided in Appendix 6C (page 16), 

show the school’s first year staffing plan, including teachers and educational support 
staff. Provide license requirements and roles for each of the staff members you 
provide in your plan. In one paragraph explain how your staffing plan reflects your 
educational priorities. 

d. Staff Handbook: When the school is at full capacity, you may choose to develop a 

handbook for staff, parents, and community partners that describes the governance 
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structure of the school. Write an extract (one page) from such a handbook that 
explains one critical aspect of the governance structure of the school. 

Part C: Evidence of Capacity 

Please submit the resumes and partnership agreements to the fullest extent possible. 

1 . Planning Team Capacity: Write a one page overview of the strengths of the planning team 
that serves as an introduction to the resumes. Provide a resume for each member of the 
planning team, including his/her educational and employment history. The resume and/or 
cover letter should describe the leadership capacity of the key personnel, including evidence of 
expertise and a strong track record in the following areas: 

a. Instructional leadership, including 

■ Organizing toward high student achievement 

■ Professional development 

■ Curriculum development 

■ Youth development 

b. Urban school development and operation 

c. Parent and community engagement 

d. Financial management 

e. Creation of strategic partnerships 

2. Institutional Partnerships: 

a. If the proposed school anticipates partnering with an outside entity (i.e. an 
intermediary and/or community-based organization), provide evidence of the 
organization’s track record in terms of a. - g. above. 

b. Describe the role that the partner(s) would play in the school’s educational plan and 
operation. 

3. Leadership Capacity: 

a. Attach a copy of the proposed school leader’s SAS/SBL or SDA/SDL certificate or 
proof of enrollment in a New York State principal certification program, indicating license 
date. 

b. Attach the names, positions and contact information of three potential referees who 
will testify to the integrity and leadership qualities of the proposed school leader. 

Part D: Connection to Community 

Provide letters of commitment from institutional and/or community based partners that will 
support the school. These letters must be on official letterhead and should explain the specific 
roles of each partner in the school. Fiduciary and governance structures should also be 
explained. 
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Stage 3: The Interview, December 10-14, 2007 



Planning teams with the most developed applications will be invited to interview with 
representatives of the New City Department of Education during the week of December 10-14, 
2007. Separate notification and explanation of the interview format will be sent. 
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Signatures 



I certify that the work in this application portfolio is the collaborative work of the planning 
team for 

Name of School 



Attach this page to the original application portfolio and submit to the Office of New 
Schools by 5pm on November 28, 2007 



Print/Type Name of Proposed School Leader Signature Date 



Print/Type Name of Intermediary Organization Executive Director Signature Date 



Print/Type Name of Leadership Program Advisor who will certify that Principal Candidate will 
meet all license requirements by July 1, 2008 OR Attach copy of SDA/SDL or SAS/SBL 
certification 
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Attachment 2A: School Directory Page 


Important Admissions Information 


School Overview 


Eligibility: (Choose One) 


In Their Own Words: (500 Characters Maximum) 


□ Limited Unscreened: Priority to students 
who attend an information session, then to 
all New York City Residents 

□ Other: 




Special Education Services (Check all 
the apply): 

M SETTS 

□ Self-contained 

□ Collaborative Team Teaching 

□ Special Class for Hearing Impaired 


Partnerships 

■ Intermediary: 

■ Community-based Organizations: 

■ Hospital Outreach: 

■ Cultural/Arts Organizations: 

■ Not-For-Profit: 

■ Corporate: 

■ Financial Institutions: 

■ Other: 


ELL Programs (Check all that apply): 


Courses & Program Highlights 


M ESL 

□ Bilingual Programs: 

□ Dual Language Programs: 


Programs: 


Enrollment: 


Languages: 


Total Students per grade: 
Grades Served in 2008-2009: 
Grades Served at Scale: 


Advanced Placement Courses: 




Extracurricular Activities: 




Leadership & Support: 
Academic: 

Artistic: 

Clubs: 

School Sports: 
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Attachment 6C: Staffing Plan Guide 



Use the following hypothetical numbers as a guide to create a staffing plan that will enable the 
school to execute its mission. Provide the specific license and role of the staff member in the 
school. 

Important: These figures are provided only as a guide and should not create an expectation for 
the 2008 budget. The purpose of this application question is to create a staffing plan, and so 
this budget does not include start-up allocation, OTPS expenditures, etc. Official school 
budgets, released in the spring of 2008, will provide more specific allocations. 



salary assumptions 


Please use the following in your budget 


Principal 


$107,000 


School Secretary 


$43,300 


Social Worker 


$65,000 


Guidance Counselor 


$76,000 


Teacher 


$55,000 


y 2 Time F Status Teacher 


$27,500 


Paraprofessional 


$33,500 


School Aide 


$23,500 



ALLOCATIONS: Use these hypothetical allocations to create school’s first year staffing plan 




9-12 

beginning with 
108 Students 


6-12 

beginning with 
81 Students 


6-12 

beginning with 
162 Students 


6-8 

beginning with 
108 Students 


Overall Allocation 


$615,000 


$615,000 


$837,000 


$615,000 



NOTE: 

The Overall Allocation for the purposes of this application includes overhead and instructional allocations 
and can be used to fund any of the following positions: principal, secretary, guidance counselor, social 
worker, school aide, paraprofessional, and teacher. It is not necessary to provide budget for OTPS. 
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